Nov 02 21:10:00.426142 ip-10-0-133-33 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 21:10:00.426152 ip-10-0-133-33 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 21:10:00.426159 ip-10-0-133-33 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 21:10:00.426373 ip-10-0-133-33 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 21:10:11.826318 ip-10-0-133-33 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 21:10:11.826336 ip-10-0-133-33 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d7709a173d944e728828fda4e2c844b6 -- Nov 02 21:12:12.305821 ip-10-0-133-33 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:12:12.640368 ip-10-0-133-33 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:12.640368 ip-10-0-133-33 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:12:12.640368 ip-10-0-133-33 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:12.640368 ip-10-0-133-33 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:12.644121 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644042 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.644121 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644121 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.644121 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644125 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644129 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644132 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644136 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644144 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644148 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644151 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644154 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644157 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644160 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644162 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644165 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644167 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644171 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644173 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644176 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644178 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644181 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644184 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.644214 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644186 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644189 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644191 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644194 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644197 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644199 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644202 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644205 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644207 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644210 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644213 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644215 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644218 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644221 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644224 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644227 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644230 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644232 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644235 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644238 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.644678 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644241 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644243 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644245 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644248 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644251 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644253 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644256 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644258 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644261 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644263 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644266 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644268 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644271 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644273 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644276 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644279 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644282 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644285 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644287 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644290 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.645507 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644292 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644297 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644302 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644305 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644308 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644310 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644313 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644316 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644319 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644321 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644324 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644327 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644329 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644332 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644334 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644337 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644339 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644342 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644344 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.646049 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644347 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644349 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644352 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644355 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644358 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644360 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644363 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644366 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644368 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644371 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644373 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644376 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644378 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644381 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644384 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644386 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644389 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644391 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644394 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644396 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.646505 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644399 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644401 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644405 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644407 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644410 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.644413 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646757 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646770 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646774 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646777 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646780 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646783 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646785 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646789 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646791 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646795 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646798 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646800 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646803 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646805 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.647000 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646808 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646811 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646813 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646816 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646819 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646821 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646824 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646826 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646829 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646831 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646834 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646836 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646839 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646842 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646844 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646847 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646850 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646853 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646856 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.647472 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646859 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646862 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646865 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646867 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646870 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646874 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646878 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646881 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646885 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646887 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646890 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646894 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646898 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646902 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646905 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646907 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646910 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646913 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646916 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.647936 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646919 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646921 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646924 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646926 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646929 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646932 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646934 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646937 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646939 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646943 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646946 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646948 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646951 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646954 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646957 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646960 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646962 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646965 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646967 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646970 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.648379 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646972 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646975 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646977 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646980 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646982 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646985 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646988 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646990 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646993 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646996 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.646998 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647001 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647004 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647007 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647010 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647013 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647015 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647018 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647020 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.648895 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647023 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647025 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647028 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647031 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647034 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647036 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647040 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647042 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647045 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647047 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647050 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647052 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647055 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647057 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647061 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647581 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647588 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647591 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647594 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647597 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.649337 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647599 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647603 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647606 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647608 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647611 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647614 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647617 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647619 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647622 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647625 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647628 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647631 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647633 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647636 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647639 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647642 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647644 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647647 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647650 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.649811 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647652 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647655 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647658 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647660 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647663 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647666 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647668 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647671 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647673 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647676 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647678 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647681 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647684 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647686 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647690 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647692 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647695 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647698 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647701 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.650265 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647703 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647706 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647709 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647711 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647714 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647716 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647719 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647721 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647724 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647727 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647729 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647732 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647734 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647737 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647740 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647742 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647745 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647747 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647750 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647752 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.650733 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647755 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647757 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647759 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647762 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647764 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647769 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647773 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647778 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647781 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647784 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647787 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647790 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647793 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647796 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647799 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647802 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647804 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647806 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647809 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.651223 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647811 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647815 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647817 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647820 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647822 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647825 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647827 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647830 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647832 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647834 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647837 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647839 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647842 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647844 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647847 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647849 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647852 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647854 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647857 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.651689 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647860 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647862 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647865 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647867 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.647869 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648674 2581 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648686 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648694 2581 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648699 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648704 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648708 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648712 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648717 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648720 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648723 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648726 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648730 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648733 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648736 2581 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648739 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648742 2581 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648745 2581 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648748 2581 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648752 2581 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:12:12.652129 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648755 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648758 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648761 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648765 2581 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648769 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648772 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648775 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648779 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648783 2581 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648787 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648790 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648793 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648796 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648806 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648809 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648812 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648816 2581 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648819 2581 flags.go:64] FLAG: --enable-server="true" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648823 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648827 2581 flags.go:64] FLAG: --event-burst="100" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648831 2581 flags.go:64] FLAG: --event-qps="50" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648834 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648837 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648840 2581 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:12:12.652690 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648845 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648848 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648851 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648854 2581 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648857 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648860 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648863 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648866 2581 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648869 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648872 2581 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648875 2581 flags.go:64] FLAG: --feature-gates="" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648879 2581 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648882 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648886 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648889 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648892 2581 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648895 2581 flags.go:64] FLAG: --help="false" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648898 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648902 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648905 2581 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648908 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648912 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648915 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:12:12.653295 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648918 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648921 2581 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648925 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648928 2581 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648931 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648935 2581 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648937 2581 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648940 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648943 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648946 2581 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648949 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648952 2581 flags.go:64] FLAG: --lock-file="" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648954 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648957 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648961 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648966 2581 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648969 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648972 2581 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648975 2581 flags.go:64] FLAG: --logging-format="text" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648978 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648981 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648984 2581 flags.go:64] FLAG: --manifest-url="" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648987 2581 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648991 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648994 2581 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:12:12.653849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.648998 2581 flags.go:64] FLAG: --max-pods="110" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649001 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649004 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649007 2581 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649010 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649013 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649016 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649019 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649027 2581 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649034 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649041 2581 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649044 2581 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649047 2581 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649050 2581 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649054 2581 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649057 2581 flags.go:64] FLAG: --port="10250" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649060 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649062 2581 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-072f91109dd8c80ad" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649065 2581 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649069 2581 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649072 2581 flags.go:64] FLAG: --register-node="true" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649074 2581 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649079 2581 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649081 2581 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649084 2581 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649087 2581 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:12:12.654454 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649091 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649094 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649097 2581 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649100 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649103 2581 flags.go:64] FLAG: --runonce="false" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649106 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649109 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649112 2581 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649115 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649118 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649121 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649124 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649127 2581 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649130 2581 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649133 2581 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649138 2581 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649141 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649145 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649148 2581 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649150 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649156 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649159 2581 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649162 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649166 2581 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649169 2581 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:12:12.655155 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649172 2581 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649174 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649178 2581 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649180 2581 flags.go:64] FLAG: --v="2" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649184 2581 flags.go:64] FLAG: --version="false" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649189 2581 flags.go:64] FLAG: --vmodule="" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649192 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649196 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649333 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649337 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649341 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649345 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649350 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649352 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649355 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649358 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649361 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649363 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649366 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649369 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649371 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.655773 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649374 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649378 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649380 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649383 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649386 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649389 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649392 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649394 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649397 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649400 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649402 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649405 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649407 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649410 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649413 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649416 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649419 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649421 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649424 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.656274 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649426 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649429 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649432 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649435 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649439 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649441 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649444 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649446 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649449 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649451 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649454 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649456 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649459 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649461 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649465 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649467 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649470 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649473 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649476 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649478 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.656748 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649481 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649483 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649486 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649489 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649491 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649494 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649496 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649499 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649502 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649504 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649507 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649509 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649512 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649514 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649528 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649531 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649534 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649537 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649540 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649542 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.657217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649545 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649548 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649550 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649552 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649555 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649557 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649561 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649565 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649568 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649571 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649574 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649577 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649581 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649583 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649586 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649588 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649591 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649593 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649596 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649598 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.657732 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649601 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649603 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649606 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649608 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649611 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649614 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649616 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649619 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649623 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649625 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649628 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649631 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649633 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649636 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.649691 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649730 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.658217 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649734 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649737 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649740 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649743 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649746 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649749 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649751 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649754 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649758 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649761 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649764 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649767 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649770 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649772 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649775 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649777 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649780 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649782 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649785 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649788 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.658605 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649790 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649792 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649795 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649797 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649801 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649804 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649806 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649809 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649812 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649815 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649817 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649820 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649823 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649825 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649828 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649831 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649834 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649836 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649839 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649841 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.659076 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649844 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649847 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649850 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649852 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649855 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649858 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649860 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649862 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649865 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649867 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649870 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649872 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649875 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649877 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649880 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649882 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649887 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649889 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649892 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649894 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.659548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649896 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649900 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649904 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649906 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649909 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649913 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649915 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649918 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649921 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649923 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649926 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649928 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649931 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649933 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649936 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649939 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649942 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649944 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649947 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649949 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.660030 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649952 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649954 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649957 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649959 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649962 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649964 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649967 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649970 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649973 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649976 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649981 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649985 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649987 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649990 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649992 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649995 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.649997 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650000 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650002 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.660492 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650005 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650008 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650010 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650012 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650015 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.650017 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.656588 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.656605 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.656623 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.656628 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656683 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656688 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656692 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656695 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656697 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656700 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656702 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656706 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656708 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.660956 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656711 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656714 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656716 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656719 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656722 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656725 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656728 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656730 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656733 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656735 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656740 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656744 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656747 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656751 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656754 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656756 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656759 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656762 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656765 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.661406 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656768 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656772 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656775 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656778 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656781 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656784 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656786 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656789 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656791 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656794 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656797 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656800 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656802 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656805 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656808 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656810 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656813 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656815 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656818 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.661858 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656821 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656824 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656827 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656830 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656832 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656835 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656837 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656842 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656845 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656848 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656851 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656854 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656857 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656860 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656863 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656866 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656869 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656872 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656874 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.662316 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656877 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656879 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656882 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656885 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656887 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656890 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656892 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656895 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656897 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656900 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656903 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656905 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656908 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656910 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656913 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656915 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656918 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656920 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656923 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.662912 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656925 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656928 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656930 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656933 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656936 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656938 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656941 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656944 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656947 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656950 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656953 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656955 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656958 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656961 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656963 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656966 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656968 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656971 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656973 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656976 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.663362 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.656978 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.657004 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657148 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657159 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657162 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657164 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657167 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657170 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657172 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657175 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657177 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657180 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657184 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657188 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657191 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.663886 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657194 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657197 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657200 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657202 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657205 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657208 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657210 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657213 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657216 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657219 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657222 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657224 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657227 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657230 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657232 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657235 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657237 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657240 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657242 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657245 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.664242 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657248 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657250 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657254 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657258 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657261 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657263 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657267 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657270 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657273 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657275 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657278 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657280 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657283 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657286 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657288 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657291 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657293 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657296 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657299 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657301 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.664717 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657304 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657307 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657310 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657313 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657315 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657318 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657320 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657323 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657325 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657328 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657330 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657333 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657335 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657338 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657340 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657342 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657345 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657347 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657350 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657352 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.665191 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657355 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657357 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657360 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657362 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657365 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657367 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657370 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657372 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657375 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657377 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657380 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657383 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657385 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657387 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657390 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657393 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657395 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657398 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657400 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.665663 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657403 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657405 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657408 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657411 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657413 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657416 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657419 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657421 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657424 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657426 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657429 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657431 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657434 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657436 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.657459 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657490 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:12.666105 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657495 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657498 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657501 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657504 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657506 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657509 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657512 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657514 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657530 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657533 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657535 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657538 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657541 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657544 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657546 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657548 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657551 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657554 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657556 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:12.666488 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657559 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657561 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657564 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657567 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657569 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657572 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657574 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657577 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657579 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657582 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657584 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657587 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657589 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657592 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657594 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657598 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657602 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657605 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657607 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:12.667089 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657610 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657613 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657616 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657618 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657621 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657623 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657626 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657629 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657631 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657634 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657636 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657639 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657641 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657644 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657646 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657649 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657651 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657654 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657657 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657659 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:12.667540 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657662 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657666 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657669 2581 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657672 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657675 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657677 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657680 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657682 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657688 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657690 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657693 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657695 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657699 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657702 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657704 2581 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657707 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657710 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657712 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657715 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657717 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:12.668483 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657721 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657723 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657726 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657728 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657731 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657734 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657737 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657739 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657742 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657744 2581 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657747 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657750 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657752 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657755 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657757 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657759 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657763 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657765 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657767 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657770 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:12.668955 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657774 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657776 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657779 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657782 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657785 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657787 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:12.657790 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.657845 2581 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.660553 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.661347 2581 server.go:1014] "Starting client certificate rotation" Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.661438 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.661466 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.663305 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:12.669409 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.665946 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:12.674651 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.674627 2581 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:12:12.677071 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.677055 2581 log.go:25] "Validated CRI v1 image API" Nov 02 21:12:12.677932 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.677903 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:12:12.680087 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.680068 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9bc9ea3a-b093-4227-ab0d-d1cbc37f76b4:/dev/nvme0n1p3 e6e9f5e6-ab13-4a08-896e-9fabe2c4f7c8:/dev/nvme0n1p4] Nov 02 21:12:12.680153 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.680090 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 21:12:12.685989 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.685882 2581 manager.go:217] Machine: {Timestamp:2026-11-02 21:12:12.683943236 +0000 UTC m=+0.281042598 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108769 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21df34985c13c3e9400ca9794e2bf5 SystemUUID:ec21df34-985c-13c3-e940-0ca9794e2bf5 BootID:d7709a17-3d94-4e72-8828-fda4e2c844b6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e7:5a:2d:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e7:5a:2d:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:39:e9:fa:2c:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:12:12.686768 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.686757 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:12:12.686853 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.686835 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:12:12.689201 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.689180 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:12:12.689326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.689203 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-33.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:12:12.689789 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.689780 2581 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:12:12.689824 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.689792 2581 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:12:12.689824 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.689818 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:12.689874 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.689830 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:12.690680 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.690670 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:12:12.694352 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.694341 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:12:12.694459 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.694451 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:12:12.695025 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.695016 2581 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:12:12.695054 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.695029 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:12:12.695054 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.695038 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:12:12.695054 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.695047 2581 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:12:12.695054 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.695054 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:12:12.695371 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.695359 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:12.697254 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.697232 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:12:12.702347 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.702335 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:12.703069 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703057 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:12:12.703104 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703088 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:12:12.703245 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703235 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:12:12.703273 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703250 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:12:12.703273 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703256 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:12:12.703273 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703262 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:12:12.703273 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703269 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:12:12.703371 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703276 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:12:12.703729 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703720 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:12:12.703762 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703730 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:12:12.703762 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703736 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:12:12.703762 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703743 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:12:12.703762 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703756 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:12:12.703962 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.703764 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:12:12.704272 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.704263 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:12:12.704300 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.704273 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:12:12.708116 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.708103 2581 server.go:1263] "Started kubelet" Nov 02 21:12:12.708214 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.708169 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:12:12.708300 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.708263 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:12:12.708346 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.708316 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:12:12.709945 ip-10-0-133-33 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:12:12.711013 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.710075 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:12:12.711070 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.711023 2581 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:12:12.715055 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.714539 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:12:12.715328 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.715305 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:12:12.716053 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.716031 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:12:12.716053 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.716054 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:12:12.716220 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.716208 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:12:12.716286 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.716276 2581 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:12:12.716328 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.716286 2581 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:12:12.716710 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.716669 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:12.719907 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.719888 2581 factory.go:55] Registering systemd factory Nov 02 21:12:12.720141 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.720131 2581 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:12:12.720283 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.720092 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 21:12:12.720537 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.720505 2581 factory.go:153] Registering CRI-O factory Nov 02 21:12:12.720537 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.720538 2581 factory.go:223] Registration of the crio container factory successfully Nov 02 21:12:12.720642 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.720585 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:12:12.720642 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.720599 2581 factory.go:103] Registering Raw factory Nov 02 21:12:12.720642 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.720609 2581 manager.go:1199] Started watching for new ooms in manager Nov 02 21:12:12.721470 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.721452 2581 manager.go:319] Starting recovery of all containers Nov 02 21:12:12.731071 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.731056 2581 manager.go:324] Recovery completed Nov 02 21:12:12.735828 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.735817 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:12.737857 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.737840 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:12.737944 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.737885 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:12.737944 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.737901 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:12.738427 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.738412 2581 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:12:12.738497 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.738427 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:12:12.738497 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.738437 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:12.738497 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.738451 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:12:12.739121 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.739104 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:12.739170 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.739135 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:12.739170 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.739147 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:12.740715 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.740702 2581 policy_none.go:50] "Start" Nov 02 21:12:12.740782 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.740719 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:12:12.740782 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.740731 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:12:12.742804 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.742790 2581 policy_none.go:44] "Start" Nov 02 21:12:12.765177 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.765145 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:12:12.771248 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.771231 2581 manager.go:341] "Starting Device Plugin manager" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.771688 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.771841 2581 server.go:87] "Starting device plugin registration server" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.772106 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.772119 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.772804 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.772860 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.772883 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.772980 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.772988 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.786998 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.788098 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.788116 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.788135 2581 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:12:12.797152 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.788164 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 21:12:12.843059 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.843037 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-33.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 21:12:12.843059 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.843036 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-33.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 21:12:12.843945 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.843073 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-33.us-west-2.compute.internal.18e456bfa50c1995 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-33.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-33.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:12:12.708075925 +0000 UTC m=+0.305175286,LastTimestamp:2026-11-02 21:12:12.708075925 +0000 UTC m=+0.305175286,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-33.us-west-2.compute.internal,}" Nov 02 21:12:12.873461 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.873436 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:12.874312 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.874296 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:12.874407 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.874329 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:12.874407 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.874345 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:12.874407 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.874375 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:12.888397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.888375 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal"] Nov 02 21:12:12.888478 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.888433 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:12.889151 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.889138 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:12.889203 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.889163 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:12.889203 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.889172 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:12.891382 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.891347 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:12.891495 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.891479 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:12.891556 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.891509 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:12.892014 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.891997 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:12.892086 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.892023 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:12.892086 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.892033 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:12.892086 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.892077 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:12.892177 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.892098 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:12.892177 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.892107 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:12.894127 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.894110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:12.894188 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.894138 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:12.894777 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.894760 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:12.894849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.894786 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:12.894849 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.894796 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:12.921029 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.921013 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:12.923416 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.923401 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:12.933753 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.933727 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 21:12:12.933848 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.933776 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-33.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 21:12:12.973863 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.973843 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:12:12.973935 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:12.973902 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vwxnt" Nov 02 21:12:12.979911 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.979890 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 21:12:12.980000 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.979933 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 21:12:12.980000 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.979882 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-33.us-west-2.compute.internal.18e456bfa6d2873e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-33.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-33.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-33.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:12:12.737857342 +0000 UTC m=+0.334956710,LastTimestamp:2026-11-02 21:12:12.737857342 +0000 UTC m=+0.334956710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-33.us-west-2.compute.internal,}" Nov 02 21:12:12.991435 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:12.991412 2581 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-33.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.017795 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.017769 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.017866 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.017798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.017866 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.017815 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a52ae9f924fc84a5af3aafcc66e569a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"7a52ae9f924fc84a5af3aafcc66e569a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.118703 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.118672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.118703 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.118709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.118842 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.118726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a52ae9f924fc84a5af3aafcc66e569a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"7a52ae9f924fc84a5af3aafcc66e569a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.118842 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.118775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.118842 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.118779 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.118842 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.118814 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a52ae9f924fc84a5af3aafcc66e569a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"7a52ae9f924fc84a5af3aafcc66e569a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.152389 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.152345 2581 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-33.us-west-2.compute.internal\" not found" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.192509 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.192489 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:13.193390 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.193374 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:13.193480 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.193406 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:13.193480 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.193420 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:13.193480 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.193455 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.224098 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.224082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.227338 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.227314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.252565 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.252547 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:13.252639 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.252570 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-33.us-west-2.compute.internal\": node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.365064 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.365039 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.465909 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.465847 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.566427 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.566400 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.661255 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.661232 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 21:12:13.661669 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.661356 2581 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa4a48b711f844547b37f00e18a10669-41cfdfea4362ef04.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-33.us-west-2.compute.internal.18e456bfa6d3472e\": read tcp 10.0.133.33:48894->3.16.140.73:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-33.us-west-2.compute.internal.18e456bfa6d3472e default 5119 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-33.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-33.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-33.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 21:12:12 +0000 UTC,LastTimestamp:2026-11-02 21:12:12.874351024 +0000 UTC m=+0.471450387,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-33.us-west-2.compute.internal,}" Nov 02 21:12:13.667459 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.667443 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.767804 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.767751 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.867984 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.867955 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.908021 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.908001 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:13.968861 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:13.968840 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:13.978708 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.978675 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:07:13 +0000 UTC" deadline="2028-04-17 19:32:57.82451584 +0000 UTC" Nov 02 21:12:13.978761 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:13.978706 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12766h20m43.845814042s" Nov 02 21:12:14.014916 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.014897 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:14.015815 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.015799 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:14.015888 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.015829 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:14.015888 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.015845 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:14.069272 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.069221 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:14.114269 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.114245 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:14.169323 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.169290 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:14.269779 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.269756 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:14.368811 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.368759 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:14.417032 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.417010 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:14.422254 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:14.422220 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04c921cae53f031e25193b5cbbab217.slice/crio-a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26 WatchSource:0}: Error finding container a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26: Status 404 returned error can't find the container with id a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26 Nov 02 21:12:14.426582 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.426568 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:12:14.431497 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:14.431472 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a52ae9f924fc84a5af3aafcc66e569a.slice/crio-79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b WatchSource:0}: Error finding container 79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b: Status 404 returned error can't find the container with id 79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b Nov 02 21:12:14.477185 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.477167 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:14.478929 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.478915 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:14.534317 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.534300 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:14.642414 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.642359 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:14.696976 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.696960 2581 apiserver.go:52] "Watching apiserver" Nov 02 21:12:14.716576 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.716552 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 21:12:14.791937 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.791883 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"7a52ae9f924fc84a5af3aafcc66e569a","Type":"ContainerStarted","Data":"79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b"} Nov 02 21:12:14.792940 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.792920 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"c04c921cae53f031e25193b5cbbab217","Type":"ContainerStarted","Data":"a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26"} Nov 02 21:12:14.800208 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.800190 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:12:14.800491 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.800473 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal","openshift-image-registry/node-ca-f6955","openshift-network-diagnostics/network-check-target-fmkt5","kube-system/konnectivity-agent-wcgz4","openshift-multus/multus-ccljr","openshift-cluster-node-tuning-operator/tuned-q4ljn","openshift-dns/node-resolver-pdftq","openshift-multus/multus-additional-cni-plugins-qhkj4","openshift-network-operator/iptables-alerter-fbktz","openshift-ovn-kubernetes/ovnkube-node-p7wfq","kube-system/global-pull-secret-syncer-fdpbr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-b26fp"] Nov 02 21:12:14.804258 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.804240 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.806501 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.806486 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.806578 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.806563 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:14.808764 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.808748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:14.808851 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.808811 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:14.811009 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.810992 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:14.813299 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.813279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.815563 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.815545 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.817930 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.817912 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.820269 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.820251 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:14.820360 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.820305 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:14.822900 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.822883 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:14.825449 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.825434 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.827950 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.827933 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.828826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.828806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-system-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.828921 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.828831 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-cni-binary-copy\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.828985 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.828923 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-k8s-cni-cncf-io\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.828985 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.828957 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-hostroot\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829101 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-daemon-config\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829162 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-lib-modules\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.829162 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:14.829271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829176 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-etc-kubernetes\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829230 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrtm\" (UniqueName: \"kubernetes.io/projected/7bfd56dd-b587-42d9-83f8-6788d298b743-kube-api-access-hbrtm\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-iptables-alerter-script\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.829438 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829274 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpkz\" (UniqueName: \"kubernetes.io/projected/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-kube-api-access-rnpkz\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.829438 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829302 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6n9z\" (UniqueName: \"kubernetes.io/projected/0f56336c-5149-4436-8bdd-28b2d1b1573f-kube-api-access-k6n9z\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.829438 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829323 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-bin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829438 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829355 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-conf-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829438 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-multus-certs\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829438 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829426 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-var-lib-kubelet\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829464 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-tuned\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-kubelet\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829556 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-konnectivity-ca\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysconfig\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829644 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-run\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-sys\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829722 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-host-slash\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-netns\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.829880 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829867 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-multus\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-agent-certs\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829922 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-host\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhl65\" (UniqueName: \"kubernetes.io/projected/5ac06bad-5657-47d1-9f15-87fd58f9945f-kube-api-access-xhl65\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829966 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.829997 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-systemd\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-tmp\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-kubelet-config\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-dbus\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-modprobe-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830202 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-kubernetes\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830310 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f56336c-5149-4436-8bdd-28b2d1b1573f-hosts-file\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830342 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f56336c-5149-4436-8bdd-28b2d1b1573f-tmp-dir\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.830401 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830375 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-cnibin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.830886 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830425 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-socket-dir-parent\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.830886 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.830444 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-conf\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.857162 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.857143 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:12:14.857162 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.857154 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:12:14.857326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.857195 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z2vtc\"" Nov 02 21:12:14.863615 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.863579 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:12:14.863615 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.863589 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:12:14.863754 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.863699 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zzckx\"" Nov 02 21:12:14.865276 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.865260 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:12:14.865342 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.865323 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:12:14.865392 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.865379 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:12:14.865432 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.865384 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:12:14.865647 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.865633 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-q4vfp\"" Nov 02 21:12:14.869534 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.867407 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:14.869534 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.867558 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sdlzm\"" Nov 02 21:12:14.869534 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.867568 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:14.870026 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.870007 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:12:14.870085 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.870008 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:14.870085 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.870051 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:14.870162 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.870105 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6fz8\"" Nov 02 21:12:14.875109 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.875089 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:12:14.875109 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.875100 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:12:14.875230 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.875090 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:12:14.875230 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.875119 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c5zxb\"" Nov 02 21:12:14.917957 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.917902 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:12:14.930918 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.930894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-ovn\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.930992 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.930958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.930992 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.930975 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wq67\" (UniqueName: \"kubernetes.io/projected/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-kube-api-access-9wq67\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.931106 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-sys-fs\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.931106 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.931187 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931120 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-netns\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.931187 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931159 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-netns\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.931187 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-host\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhl65\" (UniqueName: \"kubernetes.io/projected/5ac06bad-5657-47d1-9f15-87fd58f9945f-kube-api-access-xhl65\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931221 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931228 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-host\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-systemd\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931251 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-tmp\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p944v\" (UniqueName: \"kubernetes.io/projected/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-kube-api-access-p944v\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931250 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931317 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-systemd\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931331 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.931332 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.931433 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret podName:ba030b9f-ec13-45a5-82a9-df4315256051 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:15.431388444 +0000 UTC m=+3.028487833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret") pod "global-pull-secret-syncer-fdpbr" (UID: "ba030b9f-ec13-45a5-82a9-df4315256051") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-slash\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-modprobe-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931545 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f56336c-5149-4436-8bdd-28b2d1b1573f-hosts-file\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f56336c-5149-4436-8bdd-28b2d1b1573f-tmp-dir\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931596 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-script-lib\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931612 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-modprobe-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931619 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f56336c-5149-4436-8bdd-28b2d1b1573f-hosts-file\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cnibin\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-conf\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-cni-binary-copy\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931695 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-k8s-cni-cncf-io\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931812 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-k8s-cni-cncf-io\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f84f0d81-e665-4b14-9680-c2dbae153f43-serviceca\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.931878 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-config\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931936 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f56336c-5149-4436-8bdd-28b2d1b1573f-tmp-dir\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.931936 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-conf\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-lib-modules\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-etc-kubernetes\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932136 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-etc-kubernetes\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrtm\" (UniqueName: \"kubernetes.io/projected/7bfd56dd-b587-42d9-83f8-6788d298b743-kube-api-access-hbrtm\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932189 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-lib-modules\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932224 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-cni-binary-copy\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-etc-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932332 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-log-socket\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-env-overrides\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kff\" (UniqueName: \"kubernetes.io/projected/c66fda8b-8c6a-480a-9f42-656b485bc49a-kube-api-access-59kff\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932460 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-iptables-alerter-script\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.932669 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932493 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6n9z\" (UniqueName: \"kubernetes.io/projected/0f56336c-5149-4436-8bdd-28b2d1b1573f-kube-api-access-k6n9z\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932519 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-bin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-node-log\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-socket-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-bin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-registration-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-tuned\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-kubelet\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932807 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-systemd-units\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-system-cni-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932865 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-kubelet\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932885 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-device-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932943 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-konnectivity-ca\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.932979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysconfig\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933051 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-iptables-alerter-script\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933069 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysconfig\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.933410 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-run\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-sys\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-host-slash\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-multus\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933203 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-run\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933211 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f84f0d81-e665-4b14-9680-c2dbae153f43-host\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933226 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-sys\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933236 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-var-lib-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933254 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-host-slash\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-agent-certs\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933282 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-multus\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933303 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933330 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-bin\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933352 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-netd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-konnectivity-ca\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933408 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-kubelet-config\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.934271 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933440 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-kubelet\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933493 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-kubelet-config\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933574 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-hostroot\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-dbus\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-kubernetes\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933696 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-hostroot\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-cnibin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-socket-dir-parent\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933767 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-dbus\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933778 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-kubernetes\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933806 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-system-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933832 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-cnibin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933846 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-socket-dir-parent\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-daemon-config\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933884 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-system-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.934844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933898 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-systemd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933923 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtstl\" (UniqueName: \"kubernetes.io/projected/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kube-api-access-dtstl\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.933976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdkb\" (UniqueName: \"kubernetes.io/projected/f84f0d81-e665-4b14-9680-c2dbae153f43-kube-api-access-hjdkb\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-netns\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpkz\" (UniqueName: \"kubernetes.io/projected/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-kube-api-access-rnpkz\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934142 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-conf-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-multus-certs\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934219 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-conf-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934251 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-var-lib-kubelet\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934325 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-daemon-config\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934371 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-multus-certs\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.935397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.934381 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-var-lib-kubelet\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.935837 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.935469 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-tmp\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.935837 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.935502 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-tuned\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.935837 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.935699 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-agent-certs\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:14.960927 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.960907 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:12:14.960985 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.960957 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nj6wn" Nov 02 21:12:14.966085 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966068 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:12:14.966277 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966260 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:12:14.966381 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966260 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:12:14.966381 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966347 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:12:14.966504 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966377 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:12:14.966504 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966380 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:12:14.966504 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966391 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:12:14.966504 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966415 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ps8jp\"" Nov 02 21:12:14.966504 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966444 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dtndn\"" Nov 02 21:12:14.966746 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966633 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:12:14.966746 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966650 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:12:14.966746 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966666 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:12:14.966746 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966667 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-crvzm\"" Nov 02 21:12:14.966746 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.966741 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:12:14.986588 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.986569 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:14.986588 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.986585 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:14.986702 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.986594 2581 projected.go:196] Error preparing data for projected volume kube-api-access-hmql8 for pod openshift-network-diagnostics/network-check-target-fmkt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:14.986702 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:14.986653 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8 podName:ce6513b3-2ba0-40fd-b5a3-517dfd17327b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:15.486637155 +0000 UTC m=+3.083736520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hmql8" (UniqueName: "kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8") pod "network-check-target-fmkt5" (UID: "ce6513b3-2ba0-40fd-b5a3-517dfd17327b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:14.988059 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.988028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhl65\" (UniqueName: \"kubernetes.io/projected/5ac06bad-5657-47d1-9f15-87fd58f9945f-kube-api-access-xhl65\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:14.989402 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.989380 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrtm\" (UniqueName: \"kubernetes.io/projected/7bfd56dd-b587-42d9-83f8-6788d298b743-kube-api-access-hbrtm\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:14.989587 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.989573 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6n9z\" (UniqueName: \"kubernetes.io/projected/0f56336c-5149-4436-8bdd-28b2d1b1573f-kube-api-access-k6n9z\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:14.989908 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:14.989893 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpkz\" (UniqueName: \"kubernetes.io/projected/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-kube-api-access-rnpkz\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:15.035581 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035649 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-bin\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035649 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-netd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035649 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.035649 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-kubelet\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035696 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-netd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-systemd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035721 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-bin\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035725 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-kubelet\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035704 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035747 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-systemd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtstl\" (UniqueName: \"kubernetes.io/projected/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kube-api-access-dtstl\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.035826 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdkb\" (UniqueName: \"kubernetes.io/projected/f84f0d81-e665-4b14-9680-c2dbae153f43-kube-api-access-hjdkb\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-netns\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-ovn\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wq67\" (UniqueName: \"kubernetes.io/projected/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-kube-api-access-9wq67\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035971 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-netns\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-sys-fs\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.035989 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-ovn\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036049 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-sys-fs\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036147 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036170 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p944v\" (UniqueName: \"kubernetes.io/projected/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-kube-api-access-p944v\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036199 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-slash\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-script-lib\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036262 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cnibin\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.036318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036279 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f84f0d81-e665-4b14-9680-c2dbae153f43-serviceca\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-slash\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036300 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036327 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cnibin\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036333 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036370 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036387 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-config\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036419 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036491 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-etc-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-log-socket\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036603 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-env-overrides\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kff\" (UniqueName: \"kubernetes.io/projected/c66fda8b-8c6a-480a-9f42-656b485bc49a-kube-api-access-59kff\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036636 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-etc-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:15.037192 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-node-log\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036716 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-socket-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036751 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-registration-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-systemd-units\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-system-cni-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-device-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f84f0d81-e665-4b14-9680-c2dbae153f43-host\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-var-lib-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036906 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-log-socket\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036937 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-var-lib-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036938 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-registration-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036965 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-node-log\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036975 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-system-cni-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.036983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-systemd-units\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037006 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f84f0d81-e665-4b14-9680-c2dbae153f43-host\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.037049 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037084 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-device-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.037830 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.037097 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:15.537079943 +0000 UTC m=+3.134179313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:15.038326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.038326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037626 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-socket-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.038326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037673 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-script-lib\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.038326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037694 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f84f0d81-e665-4b14-9680-c2dbae153f43-serviceca\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:15.038326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037774 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-env-overrides\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.038326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037871 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.038326 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.037892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-config\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.092776 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.092747 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtstl\" (UniqueName: \"kubernetes.io/projected/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kube-api-access-dtstl\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.092978 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.092958 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wq67\" (UniqueName: \"kubernetes.io/projected/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-kube-api-access-9wq67\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.093127 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.093107 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p944v\" (UniqueName: \"kubernetes.io/projected/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-kube-api-access-p944v\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:15.093608 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.093577 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdkb\" (UniqueName: \"kubernetes.io/projected/f84f0d81-e665-4b14-9680-c2dbae153f43-kube-api-access-hjdkb\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:15.093899 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.093881 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kff\" (UniqueName: \"kubernetes.io/projected/c66fda8b-8c6a-480a-9f42-656b485bc49a-kube-api-access-59kff\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.115161 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.115145 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:15.122662 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.122647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:15.125162 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.125142 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f56336c_5149_4436_8bdd_28b2d1b1573f.slice/crio-3ac38e865a9cbfaed57dac6b82064cc581ad5765292a96c1eedc071ff76fd6dc WatchSource:0}: Error finding container 3ac38e865a9cbfaed57dac6b82064cc581ad5765292a96c1eedc071ff76fd6dc: Status 404 returned error can't find the container with id 3ac38e865a9cbfaed57dac6b82064cc581ad5765292a96c1eedc071ff76fd6dc Nov 02 21:12:15.130700 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.130681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ccljr" Nov 02 21:12:15.132033 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.132015 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08981c9_b9bb_4f2a_aa0e_30a1be35d954.slice/crio-528a47cb352e2868f8f32e37aa9878d8767224397a3e1a0e3764dc0624a81890 WatchSource:0}: Error finding container 528a47cb352e2868f8f32e37aa9878d8767224397a3e1a0e3764dc0624a81890: Status 404 returned error can't find the container with id 528a47cb352e2868f8f32e37aa9878d8767224397a3e1a0e3764dc0624a81890 Nov 02 21:12:15.135691 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.135673 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:15.139545 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.139528 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfd56dd_b587_42d9_83f8_6788d298b743.slice/crio-5594eef07663150f8b75062d60b418657fd5e26f1fbe38b8bea22b16dcb27266 WatchSource:0}: Error finding container 5594eef07663150f8b75062d60b418657fd5e26f1fbe38b8bea22b16dcb27266: Status 404 returned error can't find the container with id 5594eef07663150f8b75062d60b418657fd5e26f1fbe38b8bea22b16dcb27266 Nov 02 21:12:15.141715 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.141698 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:15.143548 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.143526 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac06bad_5657_47d1_9f15_87fd58f9945f.slice/crio-fc99edcce07a3309b18f8b4072ae063602b23b3f7b23033c396dae9545ebb677 WatchSource:0}: Error finding container fc99edcce07a3309b18f8b4072ae063602b23b3f7b23033c396dae9545ebb677: Status 404 returned error can't find the container with id fc99edcce07a3309b18f8b4072ae063602b23b3f7b23033c396dae9545ebb677 Nov 02 21:12:15.148772 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.148750 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:15.153390 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.153372 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bb07a2_a355_485e_a0f3_b48d5d8f9862.slice/crio-7975e295f560a9928016070bddf5aeb7e767dbeef3e41a945daadd5bb2b6ccf9 WatchSource:0}: Error finding container 7975e295f560a9928016070bddf5aeb7e767dbeef3e41a945daadd5bb2b6ccf9: Status 404 returned error can't find the container with id 7975e295f560a9928016070bddf5aeb7e767dbeef3e41a945daadd5bb2b6ccf9 Nov 02 21:12:15.158106 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.158084 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84f0d81_e665_4b14_9680_c2dbae153f43.slice/crio-498871e6112d2b739fc8bf7de678d7079989ffd9fdaeb34a5d4194786f49f365 WatchSource:0}: Error finding container 498871e6112d2b739fc8bf7de678d7079989ffd9fdaeb34a5d4194786f49f365: Status 404 returned error can't find the container with id 498871e6112d2b739fc8bf7de678d7079989ffd9fdaeb34a5d4194786f49f365 Nov 02 21:12:15.182652 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.182598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:15.184864 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.184842 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:15.191049 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.191032 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:15.192914 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.192893 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66fda8b_8c6a_480a_9f42_656b485bc49a.slice/crio-779b73cb05a4b93cd8fb22397823417f9c584f92466dd3f6dcfeca5a730aa854 WatchSource:0}: Error finding container 779b73cb05a4b93cd8fb22397823417f9c584f92466dd3f6dcfeca5a730aa854: Status 404 returned error can't find the container with id 779b73cb05a4b93cd8fb22397823417f9c584f92466dd3f6dcfeca5a730aa854 Nov 02 21:12:15.195899 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.195876 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4b2790_68ec_4dd1_8085_1aeb5fe3521e.slice/crio-29cde747bf78d802fdba65f7ce7ddec8cf02e20ae76cd61f421bf2c3d2619c48 WatchSource:0}: Error finding container 29cde747bf78d802fdba65f7ce7ddec8cf02e20ae76cd61f421bf2c3d2619c48: Status 404 returned error can't find the container with id 29cde747bf78d802fdba65f7ce7ddec8cf02e20ae76cd61f421bf2c3d2619c48 Nov 02 21:12:15.200293 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:15.200274 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f955fe_25c7_49e5_8c6b_a9abea532bc1.slice/crio-99faea52c621a4233bb363093e1527c278d08bd5944a54df56697d9d3cc6ab8c WatchSource:0}: Error finding container 99faea52c621a4233bb363093e1527c278d08bd5944a54df56697d9d3cc6ab8c: Status 404 returned error can't find the container with id 99faea52c621a4233bb363093e1527c278d08bd5944a54df56697d9d3cc6ab8c Nov 02 21:12:15.440657 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.440571 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:15.440801 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.440674 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:15.440801 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.440722 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret podName:ba030b9f-ec13-45a5-82a9-df4315256051 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:16.44071038 +0000 UTC m=+4.037809734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret") pod "global-pull-secret-syncer-fdpbr" (UID: "ba030b9f-ec13-45a5-82a9-df4315256051") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:15.541643 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.541577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:15.541643 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.541623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:15.541880 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.541717 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:15.541880 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.541740 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:15.541880 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.541746 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:15.541880 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.541754 2581 projected.go:196] Error preparing data for projected volume kube-api-access-hmql8 for pod openshift-network-diagnostics/network-check-target-fmkt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:15.541880 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.541792 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:16.541778022 +0000 UTC m=+4.138877372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:15.541880 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:15.541805 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8 podName:ce6513b3-2ba0-40fd-b5a3-517dfd17327b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:16.541799241 +0000 UTC m=+4.138898590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmql8" (UniqueName: "kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8") pod "network-check-target-fmkt5" (UID: "ce6513b3-2ba0-40fd-b5a3-517dfd17327b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:15.798625 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.798521 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6955" event={"ID":"f84f0d81-e665-4b14-9680-c2dbae153f43","Type":"ContainerStarted","Data":"498871e6112d2b739fc8bf7de678d7079989ffd9fdaeb34a5d4194786f49f365"} Nov 02 21:12:15.801452 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.801104 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbktz" event={"ID":"b8bb07a2-a355-485e-a0f3-b48d5d8f9862","Type":"ContainerStarted","Data":"7975e295f560a9928016070bddf5aeb7e767dbeef3e41a945daadd5bb2b6ccf9"} Nov 02 21:12:15.802712 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.802639 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" event={"ID":"5ac06bad-5657-47d1-9f15-87fd58f9945f","Type":"ContainerStarted","Data":"fc99edcce07a3309b18f8b4072ae063602b23b3f7b23033c396dae9545ebb677"} Nov 02 21:12:15.804755 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.804726 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"779b73cb05a4b93cd8fb22397823417f9c584f92466dd3f6dcfeca5a730aa854"} Nov 02 21:12:15.806634 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.806605 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ccljr" event={"ID":"7bfd56dd-b587-42d9-83f8-6788d298b743","Type":"ContainerStarted","Data":"5594eef07663150f8b75062d60b418657fd5e26f1fbe38b8bea22b16dcb27266"} Nov 02 21:12:15.808646 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.808622 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcgz4" event={"ID":"b08981c9-b9bb-4f2a-aa0e-30a1be35d954","Type":"ContainerStarted","Data":"528a47cb352e2868f8f32e37aa9878d8767224397a3e1a0e3764dc0624a81890"} Nov 02 21:12:15.812186 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.812155 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pdftq" event={"ID":"0f56336c-5149-4436-8bdd-28b2d1b1573f","Type":"ContainerStarted","Data":"3ac38e865a9cbfaed57dac6b82064cc581ad5765292a96c1eedc071ff76fd6dc"} Nov 02 21:12:15.813744 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.813709 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerStarted","Data":"99faea52c621a4233bb363093e1527c278d08bd5944a54df56697d9d3cc6ab8c"} Nov 02 21:12:15.815223 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.815199 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" event={"ID":"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e","Type":"ContainerStarted","Data":"29cde747bf78d802fdba65f7ce7ddec8cf02e20ae76cd61f421bf2c3d2619c48"} Nov 02 21:12:15.965375 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.965215 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:07:15 +0000 UTC" deadline="2028-06-05 00:43:46.082510797 +0000 UTC" Nov 02 21:12:15.965375 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:15.965370 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13923h31m30.117146036s" Nov 02 21:12:16.449240 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.449179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:16.449422 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.449410 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:16.449487 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.449475 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret podName:ba030b9f-ec13-45a5-82a9-df4315256051 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:18.449457042 +0000 UTC m=+6.046556408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret") pod "global-pull-secret-syncer-fdpbr" (UID: "ba030b9f-ec13-45a5-82a9-df4315256051") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:16.549852 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.549804 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:16.550032 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.549874 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:16.550092 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.550036 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:16.550092 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.550085 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:18.550071943 +0000 UTC m=+6.147171293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:16.550219 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.550147 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:16.550219 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.550161 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:16.550219 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.550174 2581 projected.go:196] Error preparing data for projected volume kube-api-access-hmql8 for pod openshift-network-diagnostics/network-check-target-fmkt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:16.550219 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.550214 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8 podName:ce6513b3-2ba0-40fd-b5a3-517dfd17327b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:18.550202228 +0000 UTC m=+6.147301585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmql8" (UniqueName: "kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8") pod "network-check-target-fmkt5" (UID: "ce6513b3-2ba0-40fd-b5a3-517dfd17327b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:16.792453 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.792341 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:16.792645 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.792497 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:16.792645 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.792611 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:16.792832 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.792700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:16.792832 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.792766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:16.792960 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:16.792846 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:16.965844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.965745 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:07:15 +0000 UTC" deadline="2028-05-12 19:23:24.125957107 +0000 UTC" Nov 02 21:12:16.965844 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:16.965843 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13366h11m7.160120114s" Nov 02 21:12:18.468352 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:18.468273 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:18.468936 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.468458 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:18.468936 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.468528 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret podName:ba030b9f-ec13-45a5-82a9-df4315256051 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:22.468509244 +0000 UTC m=+10.065608620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret") pod "global-pull-secret-syncer-fdpbr" (UID: "ba030b9f-ec13-45a5-82a9-df4315256051") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:18.569318 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:18.569278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:18.569524 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:18.569331 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:18.569524 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.569501 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:18.569664 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.569585 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:22.5695467 +0000 UTC m=+10.166646067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:18.569758 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.569692 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:18.569758 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.569713 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:18.569758 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.569726 2581 projected.go:196] Error preparing data for projected volume kube-api-access-hmql8 for pod openshift-network-diagnostics/network-check-target-fmkt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:18.569921 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.569767 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8 podName:ce6513b3-2ba0-40fd-b5a3-517dfd17327b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:22.569753766 +0000 UTC m=+10.166853129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmql8" (UniqueName: "kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8") pod "network-check-target-fmkt5" (UID: "ce6513b3-2ba0-40fd-b5a3-517dfd17327b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:18.789778 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:18.789708 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:18.800116 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:18.796685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:18.800116 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.796877 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:18.800116 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:18.796980 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:18.800116 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.797154 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:18.800116 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:18.797317 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:19.852720 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:19.852678 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"7a52ae9f924fc84a5af3aafcc66e569a","Type":"ContainerStarted","Data":"b4d10830160ddd7a7c8200f197935d03752459254f35fe7120ca469687be9b8b"} Nov 02 21:12:19.860427 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:19.860373 2581 generic.go:356] "Generic (PLEG): container finished" podID="c04c921cae53f031e25193b5cbbab217" containerID="a4968370a0eaa7963000d4642cf91324112b294cd790440f7d3e2d9ec7bfe247" exitCode=0 Nov 02 21:12:19.860628 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:19.860461 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"c04c921cae53f031e25193b5cbbab217","Type":"ContainerDied","Data":"a4968370a0eaa7963000d4642cf91324112b294cd790440f7d3e2d9ec7bfe247"} Nov 02 21:12:19.865147 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:19.865118 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="397fb892b7df4149935fb1b8e29260ee2528acd911377ab4fe7772a1490002d5" exitCode=0 Nov 02 21:12:19.865280 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:19.865171 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerDied","Data":"397fb892b7df4149935fb1b8e29260ee2528acd911377ab4fe7772a1490002d5"} Nov 02 21:12:20.134137 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:20.134055 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" podStartSLOduration=5.134033739 podStartE2EDuration="5.134033739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:20.016754292 +0000 UTC m=+7.613853703" watchObservedRunningTime="2026-11-02 21:12:20.134033739 +0000 UTC m=+7.731133114" Nov 02 21:12:20.788758 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:20.788681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:20.788916 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:20.788822 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:20.788916 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:20.788681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:20.789041 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:20.788932 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:20.789041 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:20.789032 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:20.789140 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:20.789125 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:20.875127 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:20.875077 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"c04c921cae53f031e25193b5cbbab217","Type":"ContainerStarted","Data":"652033d98dad9fb19575b46e22586a6f85f139d36408ab988acbc4295aec7776"} Nov 02 21:12:20.879972 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:20.879908 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wcgz4" event={"ID":"b08981c9-b9bb-4f2a-aa0e-30a1be35d954","Type":"ContainerStarted","Data":"529bffaea7e447c06949c18af9526beef01cc1d8a20bb18eab560810d7939376"} Nov 02 21:12:21.103645 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:21.103520 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" podStartSLOduration=6.103498902 podStartE2EDuration="6.103498902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:20.990299022 +0000 UTC m=+8.587398400" watchObservedRunningTime="2026-11-02 21:12:21.103498902 +0000 UTC m=+8.700598279" Nov 02 21:12:21.104570 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:21.103865 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wcgz4" podStartSLOduration=1.9856392189999998 podStartE2EDuration="7.103851297s" totalImagesPullingTime="5.118212078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:21.103757202 +0000 UTC m=+8.700856578" watchObservedRunningTime="2026-11-02 21:12:21.103851297 +0000 UTC m=+8.700950673" Nov 02 21:12:21.890546 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:21.890454 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6955" event={"ID":"f84f0d81-e665-4b14-9680-c2dbae153f43","Type":"ContainerStarted","Data":"b799f9e5baa3fbbfcdfc4e73a8b72496d031476a095f3ee560279a92b5cc2755"} Nov 02 21:12:22.503848 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:22.503601 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:22.503848 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.503754 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:22.503848 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.503792 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret podName:ba030b9f-ec13-45a5-82a9-df4315256051 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:30.503779603 +0000 UTC m=+18.100878968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret") pod "global-pull-secret-syncer-fdpbr" (UID: "ba030b9f-ec13-45a5-82a9-df4315256051") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:22.604167 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:22.604106 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:22.604167 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:22.604157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:22.604359 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.604289 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:22.604359 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.604341 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:30.604323592 +0000 UTC m=+18.201422957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:22.604474 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.604421 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:22.604474 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.604431 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:22.604474 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.604442 2581 projected.go:196] Error preparing data for projected volume kube-api-access-hmql8 for pod openshift-network-diagnostics/network-check-target-fmkt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:22.604474 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.604468 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8 podName:ce6513b3-2ba0-40fd-b5a3-517dfd17327b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:30.60446048 +0000 UTC m=+18.201559844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmql8" (UniqueName: "kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8") pod "network-check-target-fmkt5" (UID: "ce6513b3-2ba0-40fd-b5a3-517dfd17327b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:22.789480 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:22.789449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:22.789660 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.789574 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:22.789660 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:22.789657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:22.789789 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.789732 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:22.789789 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:22.789781 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:22.789917 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:22.789852 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:22.895652 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:22.895550 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" event={"ID":"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e","Type":"ContainerStarted","Data":"65e93e919ef3b5b4987b992c8a10fbca09a270e0da9a08ce7d3519c2b10f5da0"} Nov 02 21:12:24.794569 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:24.793900 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:24.794569 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:24.794254 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:24.794569 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:24.794337 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:24.794569 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:24.794406 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:24.794569 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:24.794453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:24.794569 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:24.794513 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:24.854015 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:24.853976 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:12:24.904831 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:24.904804 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" event={"ID":"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e","Type":"ContainerStarted","Data":"14b753845b1d051811a323ec21bd46569bce2a63b7c831b879ef2c022788e97f"} Nov 02 21:12:24.907217 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:24.906587 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ccljr" event={"ID":"7bfd56dd-b587-42d9-83f8-6788d298b743","Type":"ContainerStarted","Data":"f3eff080b35de2ab83790eb61610f53dc70e91ad5a83df9df625a0f36101a03b"} Nov 02 21:12:24.909217 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:24.908814 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pdftq" event={"ID":"0f56336c-5149-4436-8bdd-28b2d1b1573f","Type":"ContainerStarted","Data":"a75155c672df1bff67e6893b34f231a203600038f2ebffc2dce5dec8000d6eb7"} Nov 02 21:12:25.013154 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.013067 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f6955" podStartSLOduration=4.698393319 podStartE2EDuration="11.013052691s" totalImagesPullingTime="6.314659372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:22.00377853 +0000 UTC m=+9.600877904" watchObservedRunningTime="2026-11-02 21:12:25.013052691 +0000 UTC m=+12.610152107" Nov 02 21:12:25.013310 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.013238 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ccljr" podStartSLOduration=2.244810038 podStartE2EDuration="11.013231955s" totalImagesPullingTime="8.768421917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:25.012755459 +0000 UTC m=+12.609854830" watchObservedRunningTime="2026-11-02 21:12:25.013231955 +0000 UTC m=+12.610331326" Nov 02 21:12:25.117803 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.117744 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pdftq" podStartSLOduration=1.916730477 podStartE2EDuration="11.117724743s" totalImagesPullingTime="9.200994266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:25.117251661 +0000 UTC m=+12.714351035" watchObservedRunningTime="2026-11-02 21:12:25.117724743 +0000 UTC m=+12.714824113" Nov 02 21:12:25.291759 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.291677 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:25.292725 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.292704 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:25.778818 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.778653 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:12:24.853996563Z","UUID":"a0da3a4a-e617-4962-8885-62da0b405f82","Handler":null,"Name":"","Endpoint":""} Nov 02 21:12:25.780472 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.780449 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:12:25.780472 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.780475 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:12:25.913369 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.913336 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="d9ce078ff5d71798c0fac426c79ea5c4c631079fb50c30f62252781671412d00" exitCode=0 Nov 02 21:12:25.914163 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:25.914142 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerDied","Data":"d9ce078ff5d71798c0fac426c79ea5c4c631079fb50c30f62252781671412d00"} Nov 02 21:12:26.789019 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:26.788982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:26.789140 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:26.789118 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:26.789199 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:26.789191 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:26.789277 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:26.789260 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:26.789325 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:26.789314 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:26.789404 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:26.789386 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:26.917763 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:26.917680 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" event={"ID":"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e","Type":"ContainerStarted","Data":"6de898ac8ca59983e9d0ef069496fdaae15718563dd5f5335df2abd4bc370b3a"} Nov 02 21:12:26.919246 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:26.919223 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fbktz" event={"ID":"b8bb07a2-a355-485e-a0f3-b48d5d8f9862","Type":"ContainerStarted","Data":"591da670422ebadb486e366b59a08b49e38d7b0df0c73e22212d2a23d8f618e4"} Nov 02 21:12:26.920470 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:26.920451 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" event={"ID":"5ac06bad-5657-47d1-9f15-87fd58f9945f","Type":"ContainerStarted","Data":"b35a9c06ed8d837e5a1bec3f868552137979c080013ae2bb32d0c156dd9dac25"} Nov 02 21:12:27.024821 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:27.024779 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" podStartSLOduration=1.9516430809999998 podStartE2EDuration="13.024766571s" totalImagesPullingTime="11.07312349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:27.024277309 +0000 UTC m=+14.621376703" watchObservedRunningTime="2026-11-02 21:12:27.024766571 +0000 UTC m=+14.621865942" Nov 02 21:12:27.129093 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:27.129042 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" podStartSLOduration=1.67264229 podStartE2EDuration="13.129024981s" totalImagesPullingTime="11.456382691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:27.128652616 +0000 UTC m=+14.725751988" watchObservedRunningTime="2026-11-02 21:12:27.129024981 +0000 UTC m=+14.726124352" Nov 02 21:12:27.232914 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:27.232818 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fbktz" podStartSLOduration=3.979201668 podStartE2EDuration="13.232803549s" totalImagesPullingTime="9.253601881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:27.232284667 +0000 UTC m=+14.829384038" watchObservedRunningTime="2026-11-02 21:12:27.232803549 +0000 UTC m=+14.829902936" Nov 02 21:12:27.925137 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:27.925048 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="b5a2137faf117dec91c27d32843591784746967aa37d0316e3b6ccc3b6474f7a" exitCode=0 Nov 02 21:12:27.925805 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:27.925152 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerDied","Data":"b5a2137faf117dec91c27d32843591784746967aa37d0316e3b6ccc3b6474f7a"} Nov 02 21:12:28.788785 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:28.788748 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:28.788948 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:28.788800 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:28.788948 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:28.788864 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:28.788948 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:28.788912 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:28.789113 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:28.788985 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:28.789113 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:28.789082 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:29.238202 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:29.237992 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:29.238560 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:29.238322 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:12:29.238883 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:29.238862 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:29.931755 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:29.931726 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="aab8a764a9395a913b6a7a47475bfa5c525fd0b737784aca3f293dca6d720009" exitCode=0 Nov 02 21:12:29.931929 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:29.931815 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerDied","Data":"aab8a764a9395a913b6a7a47475bfa5c525fd0b737784aca3f293dca6d720009"} Nov 02 21:12:30.568466 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.568415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:30.569061 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.568592 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:30.569061 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.568665 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret podName:ba030b9f-ec13-45a5-82a9-df4315256051 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:46.568646564 +0000 UTC m=+34.165745929 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret") pod "global-pull-secret-syncer-fdpbr" (UID: "ba030b9f-ec13-45a5-82a9-df4315256051") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:12:30.669051 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.669023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:30.669216 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.669100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:30.669216 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.669200 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:12:30.669293 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.669219 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:12:30.669293 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.669228 2581 projected.go:196] Error preparing data for projected volume kube-api-access-hmql8 for pod openshift-network-diagnostics/network-check-target-fmkt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:30.669293 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.669274 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8 podName:ce6513b3-2ba0-40fd-b5a3-517dfd17327b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:46.669255875 +0000 UTC m=+34.266355236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hmql8" (UniqueName: "kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8") pod "network-check-target-fmkt5" (UID: "ce6513b3-2ba0-40fd-b5a3-517dfd17327b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:12:30.669293 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.669198 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:30.669429 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.669315 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:46.669303819 +0000 UTC m=+34.266403173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:12:30.788571 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.788539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:30.788759 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.788577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:30.788759 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.788642 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:30.788759 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.788662 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:30.788759 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.788736 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:30.788929 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:30.788836 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:30.937862 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.937839 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"d13921ec8b6088b189b9b0de1ea317e2a506932c4f983d91eb3cc8ba43a4c1de"} Nov 02 21:12:30.937938 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:30.937922 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"3b8f46af4b5258bc8dae0e0f9c02b1ec50e596b7fef8ea97ce17ea0ac0674890"} Nov 02 21:12:31.943375 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:31.943179 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"f205c324c48a419cfc92e0e774aeda7e256f43411d793458d2e5efcdf3dcdf66"} Nov 02 21:12:31.943820 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:31.943430 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"bce64e9b89b71d821b5e96eca42a8ca05193fc65b059b5d2f419fb58e142daee"} Nov 02 21:12:31.943820 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:31.943479 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"4ef3f03c5583b99f5ae42ed902113423d288b15b67b9efab672cbf4750cc6b55"} Nov 02 21:12:31.943820 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:31.943536 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"5d8bd9eaec3e3a7bb1be816f4681ce7bd731b6146fe1b73cbaaee573a539fc14"} Nov 02 21:12:32.790201 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:32.790173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:32.790346 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:32.790282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:32.790346 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:32.790309 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:32.790346 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:32.790326 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:32.790446 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:32.790359 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:32.790446 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:32.790412 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:32.947894 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:32.947859 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="47a2e5392ee5d155e022b8e28bd9e470f9f7ed482374020bdcbe3dd4661ac174" exitCode=0 Nov 02 21:12:32.948359 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:32.947937 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerDied","Data":"47a2e5392ee5d155e022b8e28bd9e470f9f7ed482374020bdcbe3dd4661ac174"} Nov 02 21:12:33.952767 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:33.952701 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"8249d3f98be3e504330dbff1109d7ca95c94eba0de6a856e1fccd1ed5f868072"} Nov 02 21:12:33.955745 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:33.955718 2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="78870204121b5651960a80803a4f947139443462776aa99adc2248714ab46551" exitCode=0 Nov 02 21:12:33.955872 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:33.955757 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerDied","Data":"78870204121b5651960a80803a4f947139443462776aa99adc2248714ab46551"} Nov 02 21:12:34.788320 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:34.788291 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:34.788320 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:34.788307 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:34.788581 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:34.788397 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:34.788581 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:34.788432 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:34.788581 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:34.788501 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:34.788737 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:34.788582 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:34.961327 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:34.961296 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" event={"ID":"f0f955fe-25c7-49e5-8c6b-a9abea532bc1","Type":"ContainerStarted","Data":"4984dcbe17cc9a01b2bdbf454009c7f5a327d98d92d08875a690bfee331ebd30"} Nov 02 21:12:35.967274 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:35.967086 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" event={"ID":"c66fda8b-8c6a-480a-9f42-656b485bc49a","Type":"ContainerStarted","Data":"1bd2d517caf4d3019e5183c7edc86c3f83d051b05961ea04da805c5e5288be8b"} Nov 02 21:12:36.213066 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.213019 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" podStartSLOduration=6.5895240489999996 podStartE2EDuration="22.213005987s" totalImagesPullingTime="15.623481938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:36.166413759 +0000 UTC m=+23.763513265" watchObservedRunningTime="2026-11-02 21:12:36.213005987 +0000 UTC m=+23.810105357" Nov 02 21:12:36.213573 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.213550 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qhkj4" podStartSLOduration=8.534729 podStartE2EDuration="22.213541888s" totalImagesPullingTime="13.678812888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:35.071374831 +0000 UTC m=+22.668474201" watchObservedRunningTime="2026-11-02 21:12:36.213541888 +0000 UTC m=+23.810641255" Nov 02 21:12:36.788298 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.788266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:36.788552 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.788412 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:36.788552 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:36.788426 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:36.788639 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:36.788551 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:36.788639 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.788617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:36.788717 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:36.788700 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:36.969536 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.969497 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:36.970001 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.969592 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:36.970001 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.969645 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:36.984747 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.984724 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:36.984917 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:36.984834 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:37.981490 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:37.981425 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fdpbr"] Nov 02 21:12:37.981957 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:37.981502 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:37.981957 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:37.981593 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:37.984240 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:37.984221 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fmkt5"] Nov 02 21:12:37.984302 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:37.984291 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:37.984404 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:37.984385 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:37.985092 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:37.985069 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b26fp"] Nov 02 21:12:37.985195 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:37.985140 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:37.985256 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:37.985243 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:39.789015 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:39.788944 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:39.789421 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:39.789025 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:39.789421 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:39.789052 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:39.789421 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:39.789162 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:39.789421 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:39.789187 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:39.789421 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:39.789256 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:41.789164 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:41.789073 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:41.789164 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:41.789109 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:41.789702 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:41.789190 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fmkt5" podUID="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" Nov 02 21:12:41.789702 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:41.789270 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdpbr" podUID="ba030b9f-ec13-45a5-82a9-df4315256051" Nov 02 21:12:41.789702 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:41.789321 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:41.789702 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:41.789385 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b26fp" podUID="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" Nov 02 21:12:43.734127 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.733984 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeReady" Nov 02 21:12:43.734489 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.734228 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:12:43.788321 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.788298 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:43.788397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.788359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:43.788397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.788388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:43.838636 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.838605 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:12:43.839167 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.839148 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:12:43.839167 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.839151 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nvsgt\"" Nov 02 21:12:43.839291 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.839156 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mnft8\"" Nov 02 21:12:43.839291 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.839174 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:12:43.839291 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.839203 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r4znh\"" Nov 02 21:12:43.839291 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.839213 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:12:43.839415 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.839401 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rrj8p"] Nov 02 21:12:43.843365 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.843343 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tmmm8"] Nov 02 21:12:43.843473 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.843458 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:43.847202 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.847181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:43.899322 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.899276 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tmmm8"] Nov 02 21:12:43.902150 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.902132 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rrj8p"] Nov 02 21:12:43.938154 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.938126 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:12:43.938154 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.938137 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:12:43.938299 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.938252 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:12:43.938337 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.938297 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:12:43.938337 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.938304 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dfd2\"" Nov 02 21:12:43.938337 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.938300 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:12:43.938430 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.938340 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jzn4w\"" Nov 02 21:12:43.967361 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.967343 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c32e61-e26d-4b32-9a55-4a6770edd655-cert\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:43.967459 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.967370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d132-0233-4799-9b13-6c5c314fe8c2-config-volume\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:43.967459 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.967399 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:43.967459 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.967413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df44d132-0233-4799-9b13-6c5c314fe8c2-tmp-dir\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:43.967611 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.967465 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dqm\" (UniqueName: \"kubernetes.io/projected/b1c32e61-e26d-4b32-9a55-4a6770edd655-kube-api-access-26dqm\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:43.967611 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:43.967573 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjpz\" (UniqueName: \"kubernetes.io/projected/df44d132-0233-4799-9b13-6c5c314fe8c2-kube-api-access-rdjpz\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.068093 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068059 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c32e61-e26d-4b32-9a55-4a6770edd655-cert\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:44.068257 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068102 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d132-0233-4799-9b13-6c5c314fe8c2-config-volume\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.068257 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.068257 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068150 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df44d132-0233-4799-9b13-6c5c314fe8c2-tmp-dir\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.068257 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068173 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dqm\" (UniqueName: \"kubernetes.io/projected/b1c32e61-e26d-4b32-9a55-4a6770edd655-kube-api-access-26dqm\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:44.068257 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjpz\" (UniqueName: \"kubernetes.io/projected/df44d132-0233-4799-9b13-6c5c314fe8c2-kube-api-access-rdjpz\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.068500 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:44.068277 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:44.068500 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:44.068339 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:44.568321252 +0000 UTC m=+32.165420600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:44.068602 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068515 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df44d132-0233-4799-9b13-6c5c314fe8c2-tmp-dir\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.068796 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.068774 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d132-0233-4799-9b13-6c5c314fe8c2-config-volume\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.071979 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.071962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c32e61-e26d-4b32-9a55-4a6770edd655-cert\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:44.121842 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.121811 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjpz\" (UniqueName: \"kubernetes.io/projected/df44d132-0233-4799-9b13-6c5c314fe8c2-kube-api-access-rdjpz\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.122012 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.121995 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dqm\" (UniqueName: \"kubernetes.io/projected/b1c32e61-e26d-4b32-9a55-4a6770edd655-kube-api-access-26dqm\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:44.160787 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.160729 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:44.535036 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.534954 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tmmm8"] Nov 02 21:12:44.543697 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:44.543668 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c32e61_e26d_4b32_9a55_4a6770edd655.slice/crio-37210afdd2441124ca45894b5478ed520e5c18b8c04b6e9cc1f15cb240c23a1d WatchSource:0}: Error finding container 37210afdd2441124ca45894b5478ed520e5c18b8c04b6e9cc1f15cb240c23a1d: Status 404 returned error can't find the container with id 37210afdd2441124ca45894b5478ed520e5c18b8c04b6e9cc1f15cb240c23a1d Nov 02 21:12:44.572269 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.572247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:44.572367 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:44.572355 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:44.572409 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:44.572400 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:45.572387422 +0000 UTC m=+33.169486784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:44.984658 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:44.984574 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tmmm8" event={"ID":"b1c32e61-e26d-4b32-9a55-4a6770edd655","Type":"ContainerStarted","Data":"37210afdd2441124ca45894b5478ed520e5c18b8c04b6e9cc1f15cb240c23a1d"} Nov 02 21:12:45.578871 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:45.578833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:45.579041 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:45.578986 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:45.579084 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:45.579047 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:47.57903246 +0000 UTC m=+35.176131812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:46.586439 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:46.586402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:46.588946 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:46.588917 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:46.687247 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:46.687215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:46.687381 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:46.687313 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:46.687421 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:46.687371 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:46.687453 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:46.687447 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:13:18.687431428 +0000 UTC m=+66.284530789 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : secret "metrics-daemon-secret" not found Nov 02 21:12:46.690230 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:46.690205 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:46.802304 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:46.802272 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:46.808393 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:46.808372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:47.201622 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.201549 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fmkt5"] Nov 02 21:12:47.222183 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:47.222154 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6513b3_2ba0_40fd_b5a3_517dfd17327b.slice/crio-cba58c8d08ee268aa059706241db88ea57b9666ce7dff64166a2b7617f9223f1 WatchSource:0}: Error finding container cba58c8d08ee268aa059706241db88ea57b9666ce7dff64166a2b7617f9223f1: Status 404 returned error can't find the container with id cba58c8d08ee268aa059706241db88ea57b9666ce7dff64166a2b7617f9223f1 Nov 02 21:12:47.253916 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.253887 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fdpbr"] Nov 02 21:12:47.262336 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:47.262301 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba030b9f_ec13_45a5_82a9_df4315256051.slice/crio-32574be367160ba67b510993d900dcc2b64b769b7fd1a309bbd519eaa909cfd3 WatchSource:0}: Error finding container 32574be367160ba67b510993d900dcc2b64b769b7fd1a309bbd519eaa909cfd3: Status 404 returned error can't find the container with id 32574be367160ba67b510993d900dcc2b64b769b7fd1a309bbd519eaa909cfd3 Nov 02 21:12:47.552387 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.552314 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4c9pl"] Nov 02 21:12:47.564675 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.564292 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.594962 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.594498 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:47.594962 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:47.594636 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:47.594962 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:47.594692 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:51.594676221 +0000 UTC m=+39.191775576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:47.612824 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.612801 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:12:47.612955 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.612899 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhk5v\"" Nov 02 21:12:47.613025 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.613013 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:12:47.659311 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.659287 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:12:47.659423 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.659327 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:12:47.660402 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.660380 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4c9pl"] Nov 02 21:12:47.695390 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.695364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fj5v\" (UniqueName: \"kubernetes.io/projected/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-api-access-2fj5v\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.695491 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.695448 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da7cd6-2547-44ec-a373-e4af9726b87b-crio-socket\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.695491 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.695472 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.695614 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.695503 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da7cd6-2547-44ec-a373-e4af9726b87b-data-volume\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.695657 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.695623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.739495 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.739460 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7"] Nov 02 21:12:47.745183 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.745159 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" Nov 02 21:12:47.796133 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.796105 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fj5v\" (UniqueName: \"kubernetes.io/projected/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-api-access-2fj5v\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.796289 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.796166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da7cd6-2547-44ec-a373-e4af9726b87b-crio-socket\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.796289 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.796206 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.796289 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.796235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da7cd6-2547-44ec-a373-e4af9726b87b-data-volume\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.796289 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.796257 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.796477 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:47.796324 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:47.796477 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.796359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da7cd6-2547-44ec-a373-e4af9726b87b-crio-socket\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.796477 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:47.796406 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:48.296386821 +0000 UTC m=+35.893486185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : configmap "kube-rbac-proxy" not found Nov 02 21:12:47.796652 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.796613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da7cd6-2547-44ec-a373-e4af9726b87b-data-volume\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.796837 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:47.796789 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:47.796934 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:47.796913 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:48.296900523 +0000 UTC m=+35.893999871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:47.799747 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.799724 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-szkkx-zrfdx"] Nov 02 21:12:47.813479 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.813422 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:12:47.813479 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.813439 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:47.815055 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.815035 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7"] Nov 02 21:12:47.815168 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.815147 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.824218 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.824201 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tjjtb\"" Nov 02 21:12:47.849149 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.849129 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fj5v\" (UniqueName: \"kubernetes.io/projected/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-api-access-2fj5v\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:47.858732 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.858708 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-szkkx-zrfdx"] Nov 02 21:12:47.897450 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.897428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkbv\" (UniqueName: \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.897574 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.897458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.897574 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.897488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.897574 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.897505 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtp6\" (UniqueName: \"kubernetes.io/projected/ef5d3dd9-d000-43dc-bf45-fbae72cdd767-kube-api-access-9xtp6\") pod \"migrator-66df5d496b-tfnj7\" (UID: \"ef5d3dd9-d000-43dc-bf45-fbae72cdd767\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" Nov 02 21:12:47.897703 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.897585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.913828 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.913806 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 21:12:47.961187 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.961168 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 21:12:47.961276 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.961207 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lplw4\"" Nov 02 21:12:47.992041 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.992009 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fmkt5" event={"ID":"ce6513b3-2ba0-40fd-b5a3-517dfd17327b","Type":"ContainerStarted","Data":"cba58c8d08ee268aa059706241db88ea57b9666ce7dff64166a2b7617f9223f1"} Nov 02 21:12:47.992884 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.992865 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fdpbr" event={"ID":"ba030b9f-ec13-45a5-82a9-df4315256051","Type":"ContainerStarted","Data":"32574be367160ba67b510993d900dcc2b64b769b7fd1a309bbd519eaa909cfd3"} Nov 02 21:12:47.998280 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.998263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkbv\" (UniqueName: \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.998344 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.998289 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.998344 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.998317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.998344 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.998332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtp6\" (UniqueName: \"kubernetes.io/projected/ef5d3dd9-d000-43dc-bf45-fbae72cdd767-kube-api-access-9xtp6\") pod \"migrator-66df5d496b-tfnj7\" (UID: \"ef5d3dd9-d000-43dc-bf45-fbae72cdd767\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" Nov 02 21:12:47.998458 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.998350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.998794 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.998773 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.998882 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.998820 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:47.999190 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:47.999172 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:48.050977 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.050949 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkbv\" (UniqueName: \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:48.051055 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.051038 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtp6\" (UniqueName: \"kubernetes.io/projected/ef5d3dd9-d000-43dc-bf45-fbae72cdd767-kube-api-access-9xtp6\") pod \"migrator-66df5d496b-tfnj7\" (UID: \"ef5d3dd9-d000-43dc-bf45-fbae72cdd767\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" Nov 02 21:12:48.080049 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.080007 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" Nov 02 21:12:48.154709 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.154682 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:48.300989 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.300857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:48.301211 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.301179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:48.301326 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:48.301312 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:48.301384 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:48.301372 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:49.301355074 +0000 UTC m=+36.898454425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : configmap "kube-rbac-proxy" not found Nov 02 21:12:48.301470 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:48.301459 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:48.301510 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:48.301497 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:49.301485567 +0000 UTC m=+36.898584917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:48.649613 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.649533 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7"] Nov 02 21:12:48.661359 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:48.660771 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5d3dd9_d000_43dc_bf45_fbae72cdd767.slice/crio-faf490672a57df24674a88352d41d036f0f777ad45a1109142796ee59755e0c2 WatchSource:0}: Error finding container faf490672a57df24674a88352d41d036f0f777ad45a1109142796ee59755e0c2: Status 404 returned error can't find the container with id faf490672a57df24674a88352d41d036f0f777ad45a1109142796ee59755e0c2 Nov 02 21:12:48.702471 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.702440 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-szkkx-zrfdx"] Nov 02 21:12:48.714781 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:48.714751 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcec323e_b13a_4bc2_b83d_246ada8bbbb3.slice/crio-b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657 WatchSource:0}: Error finding container b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657: Status 404 returned error can't find the container with id b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657 Nov 02 21:12:48.998932 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:48.998203 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tmmm8" event={"ID":"b1c32e61-e26d-4b32-9a55-4a6770edd655","Type":"ContainerStarted","Data":"72507ba2262d33cf2a9e3d1a121f453cb462e58ebe115f3a3a87a495f67aeff6"} Nov 02 21:12:49.000959 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.000932 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" event={"ID":"ef5d3dd9-d000-43dc-bf45-fbae72cdd767","Type":"ContainerStarted","Data":"faf490672a57df24674a88352d41d036f0f777ad45a1109142796ee59755e0c2"} Nov 02 21:12:49.002348 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.002315 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" event={"ID":"dcec323e-b13a-4bc2-b83d-246ada8bbbb3","Type":"ContainerStarted","Data":"b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657"} Nov 02 21:12:49.103838 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.103788 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tmmm8" podStartSLOduration=2.358748222 podStartE2EDuration="6.103772087s" totalImagesPullingTime="3.745023865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:49.103370051 +0000 UTC m=+36.700469422" watchObservedRunningTime="2026-11-02 21:12:49.103772087 +0000 UTC m=+36.700871459" Nov 02 21:12:49.310566 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.310465 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:49.310566 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.310533 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:49.310768 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:49.310621 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:49.310768 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:49.310680 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:49.310768 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:49.310685 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:51.31066858 +0000 UTC m=+38.907767938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : configmap "kube-rbac-proxy" not found Nov 02 21:12:49.310768 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:49.310714 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:51.310704272 +0000 UTC m=+38.907803636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:49.328132 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.328097 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s59nj"] Nov 02 21:12:49.356397 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.356371 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.384839 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.384814 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s59nj"] Nov 02 21:12:49.405570 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.405545 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 21:12:49.405698 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.405627 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 21:12:49.405698 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.405644 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w8spg\"" Nov 02 21:12:49.405698 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.405669 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 21:12:49.405825 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.405779 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 21:12:49.511881 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.511849 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-cabundle\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.512041 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.511890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcxv2\" (UniqueName: \"kubernetes.io/projected/acb383fe-f128-49fe-b261-c2d7f5d4300f-kube-api-access-rcxv2\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.512101 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.512050 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-key\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.612672 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.612581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxv2\" (UniqueName: \"kubernetes.io/projected/acb383fe-f128-49fe-b261-c2d7f5d4300f-kube-api-access-rcxv2\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.612817 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.612718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-key\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.612817 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.612768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-cabundle\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.613623 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.613600 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-cabundle\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.619166 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.619141 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-key\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.670937 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.670879 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxv2\" (UniqueName: \"kubernetes.io/projected/acb383fe-f128-49fe-b261-c2d7f5d4300f-kube-api-access-rcxv2\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:49.680585 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:49.680557 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:50.071940 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:50.071606 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s59nj"] Nov 02 21:12:50.079321 ip-10-0-133-33 kubenswrapper[2581]: W1102 21:12:50.079294 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb383fe_f128_49fe_b261_c2d7f5d4300f.slice/crio-1c71ff9fae033cdba98fdc64ee3320f1a3e88622c9594989f82553aa56f38f93 WatchSource:0}: Error finding container 1c71ff9fae033cdba98fdc64ee3320f1a3e88622c9594989f82553aa56f38f93: Status 404 returned error can't find the container with id 1c71ff9fae033cdba98fdc64ee3320f1a3e88622c9594989f82553aa56f38f93 Nov 02 21:12:51.013037 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:51.012999 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" event={"ID":"acb383fe-f128-49fe-b261-c2d7f5d4300f","Type":"ContainerStarted","Data":"1c71ff9fae033cdba98fdc64ee3320f1a3e88622c9594989f82553aa56f38f93"} Nov 02 21:12:51.328465 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:51.328381 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:51.328465 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:51.328444 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:51.328682 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:51.328504 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 21:12:51.328682 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:51.328577 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:51.328682 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:51.328582 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.328563073 +0000 UTC m=+42.925662437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : configmap "kube-rbac-proxy" not found Nov 02 21:12:51.328682 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:51.328613 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.328601902 +0000 UTC m=+42.925701250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:51.631459 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:51.630799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:51.631459 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:51.630986 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:51.631459 ip-10-0-133-33 kubenswrapper[2581]: E1102 21:12:51.631054 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:59.631034405 +0000 UTC m=+47.228133766 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:52.017593 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:52.017504 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" event={"ID":"ef5d3dd9-d000-43dc-bf45-fbae72cdd767","Type":"ContainerStarted","Data":"6ca49de27ce3697f3d742254f2efc7e761be6c7ec1c4b9a44761e1edea4b0481"} Nov 02 21:12:52.126941 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:52.126885 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" podStartSLOduration=2.635035553 podStartE2EDuration="5.126869077s" totalImagesPullingTime="2.491833524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:52.126485804 +0000 UTC m=+39.723585174" watchObservedRunningTime="2026-11-02 21:12:52.126869077 +0000 UTC m=+39.723968449" Nov 02 21:12:53.024420 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:53.024377 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" event={"ID":"dcec323e-b13a-4bc2-b83d-246ada8bbbb3","Type":"ContainerStarted","Data":"886cec361c7d267242dcac4e416442c57157cee3f96cb277ab02a9e8bb38ad96"} Nov 02 21:12:53.027111 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:53.027079 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fmkt5" event={"ID":"ce6513b3-2ba0-40fd-b5a3-517dfd17327b","Type":"ContainerStarted","Data":"7d83591085ac5562e150ecbbc50b63c3e29edb89ec8b58e89f4510347885cf63"} Nov 02 21:12:53.131231 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:53.131167 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" podStartSLOduration=2.549997097 podStartE2EDuration="6.131148279s" totalImagesPullingTime="3.581151182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:53.130234599 +0000 UTC m=+40.727333969" watchObservedRunningTime="2026-11-02 21:12:53.131148279 +0000 UTC m=+40.728247650" Nov 02 21:12:53.235011 ip-10-0-133-33 kubenswrapper[2581]: I1102 21:12:53.234961 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fmkt5" podStartSLOduration=34.384756575 podStartE2EDuration="39.23494572s" totalImagesPullingTime="4.850189145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:12:53.234278 +0000 UTC m=+40.831377371" watchObservedRunningTime="2026-11-02 21:12:53.23494572 +0000 UTC m=+40.832045090" Nov 02 21:12:53.941644 ip-10-0-133-33 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 21:12:53.953920 ip-10-0-133-33 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 21:12:53.954086 ip-10-0-133-33 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 21:12:53.954135 ip-10-0-133-33 systemd[1]: kubelet.service: Consumed 2.283s CPU time. Nov 02 21:12:53.967228 ip-10-0-133-33 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:12:54.068600 ip-10-0-133-33 kubenswrapper[4510]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:54.068600 ip-10-0-133-33 kubenswrapper[4510]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:12:54.068600 ip-10-0-133-33 kubenswrapper[4510]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:54.068600 ip-10-0-133-33 kubenswrapper[4510]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:12:54.070605 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070506 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.070605 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070605 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.070605 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070609 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070613 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070616 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070620 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070623 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070627 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070630 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070633 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070636 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070638 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070641 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070644 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070647 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070656 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070659 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070662 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070664 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070668 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070671 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070673 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.070714 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070676 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070679 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070681 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070684 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070687 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070689 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070692 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070695 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070697 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070700 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070703 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070705 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070708 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070710 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070713 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070716 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070719 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070722 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070725 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070728 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.071254 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070730 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070733 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070736 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070738 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070741 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070744 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070747 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070749 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070753 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070758 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070761 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070764 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070767 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070770 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070774 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070776 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070779 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070782 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070784 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070787 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.071845 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070790 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070792 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070795 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070797 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070800 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070803 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070806 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070809 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070812 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070815 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070817 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070820 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070823 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070825 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070829 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070832 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070834 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070837 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070840 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.072340 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070842 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070845 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070848 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070852 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070854 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070857 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070859 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070862 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070865 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070869 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070872 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070876 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070879 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070881 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070884 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070887 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070890 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070892 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070895 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.072832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070897 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070900 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070903 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070906 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070909 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.070912 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071769 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071781 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071786 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071799 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071804 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071809 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071813 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071818 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071822 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071827 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071831 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071834 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071837 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.073337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071840 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071844 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071846 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071849 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071857 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071860 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071863 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071866 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071868 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071871 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071874 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071876 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071879 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071882 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071884 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071887 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071889 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071892 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071895 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.073837 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071897 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071900 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071903 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071905 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071916 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071920 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071922 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071925 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071928 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071930 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071933 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071935 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071938 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071940 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071943 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071946 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071948 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071951 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071954 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.074312 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071957 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071960 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071962 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071965 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071968 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071970 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071973 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071975 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071978 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071980 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071983 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071986 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071988 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071991 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071994 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071997 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.071999 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072002 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072011 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.074789 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072016 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072019 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072021 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072024 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072027 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072029 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072032 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072036 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072038 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072041 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072043 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072046 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072049 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072052 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072056 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072061 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072064 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072067 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072070 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072073 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.075269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072075 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072078 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072080 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072083 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072086 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072088 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072091 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072094 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072097 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072099 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072102 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072104 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072112 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072117 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072119 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072122 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072673 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072680 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072685 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072688 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.075785 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072690 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072693 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072696 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072699 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072701 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072704 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072707 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072710 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072712 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072715 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072718 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072720 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072723 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072725 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072728 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072731 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072734 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072736 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072739 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072742 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.076266 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072745 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072747 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072750 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072752 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072755 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072763 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072766 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072768 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072771 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072775 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072778 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072781 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072784 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072787 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072790 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072792 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072795 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072797 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072800 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072803 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.076844 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072805 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072808 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072810 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072813 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072816 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072819 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072821 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072824 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072828 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072833 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072837 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072842 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072846 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072851 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072855 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072859 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072864 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072870 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072875 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.077337 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072879 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072884 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072890 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072893 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072896 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072899 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072903 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072916 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072920 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072925 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072929 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072934 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072938 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072943 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072947 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072954 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072961 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072972 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072978 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.077824 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072983 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072987 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072990 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072995 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.072998 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073001 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073004 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073006 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073009 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073012 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073015 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073019 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073024 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073030 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073035 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073039 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073043 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073047 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073057 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.078288 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073062 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073067 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073071 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073075 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073079 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073203 4510 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073218 4510 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073226 4510 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073231 4510 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073235 4510 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073239 4510 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073243 4510 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073247 4510 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073250 4510 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073254 4510 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073257 4510 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073263 4510 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073266 4510 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073269 4510 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073272 4510 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073275 4510 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073279 4510 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073282 4510 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073286 4510 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:12:54.078832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073290 4510 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073294 4510 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073297 4510 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073301 4510 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073305 4510 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073308 4510 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073311 4510 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073314 4510 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073318 4510 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073321 4510 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073324 4510 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073327 4510 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073330 4510 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073334 4510 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073337 4510 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073341 4510 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073344 4510 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073347 4510 flags.go:64] FLAG: --enable-server="true" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073351 4510 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073356 4510 flags.go:64] FLAG: --event-burst="100" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073360 4510 flags.go:64] FLAG: --event-qps="50" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073363 4510 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073366 4510 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073369 4510 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:12:54.079851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073378 4510 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073382 4510 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073385 4510 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073388 4510 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073391 4510 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073396 4510 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073400 4510 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073406 4510 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073413 4510 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073417 4510 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073422 4510 flags.go:64] FLAG: --feature-gates="" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073436 4510 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073440 4510 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073443 4510 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073446 4510 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073449 4510 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073453 4510 flags.go:64] FLAG: --help="false" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073456 4510 flags.go:64] FLAG: --hostname-override="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073459 4510 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073462 4510 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073466 4510 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073469 4510 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073473 4510 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:12:54.080886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073476 4510 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073479 4510 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073482 4510 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073484 4510 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073487 4510 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073490 4510 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073493 4510 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073496 4510 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073499 4510 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073504 4510 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073507 4510 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073510 4510 flags.go:64] FLAG: --lock-file="" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073512 4510 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073527 4510 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073531 4510 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073537 4510 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073540 4510 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073545 4510 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073548 4510 flags.go:64] FLAG: --logging-format="text" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073551 4510 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073555 4510 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073558 4510 flags.go:64] FLAG: --manifest-url="" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073561 4510 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073567 4510 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073570 4510 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:12:54.081892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073574 4510 flags.go:64] FLAG: --max-pods="110" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073577 4510 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073581 4510 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073584 4510 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073587 4510 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073590 4510 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073593 4510 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073596 4510 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073604 4510 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073607 4510 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073610 4510 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073613 4510 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073616 4510 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073619 4510 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073622 4510 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073625 4510 flags.go:64] FLAG: --port="10250" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073630 4510 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073633 4510 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-072f91109dd8c80ad" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073636 4510 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073639 4510 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073643 4510 flags.go:64] FLAG: --register-node="true" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073645 4510 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073649 4510 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073652 4510 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073656 4510 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073659 4510 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:12:54.083020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073663 4510 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073667 4510 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073670 4510 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073674 4510 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073677 4510 flags.go:64] FLAG: --runonce="false" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073680 4510 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073683 4510 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073686 4510 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073689 4510 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073692 4510 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073695 4510 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073699 4510 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073702 4510 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073705 4510 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073708 4510 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073711 4510 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073714 4510 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073717 4510 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073720 4510 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073724 4510 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073737 4510 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073740 4510 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073745 4510 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073749 4510 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073752 4510 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:12:54.084134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073754 4510 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073758 4510 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073761 4510 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073763 4510 flags.go:64] FLAG: --v="2" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073768 4510 flags.go:64] FLAG: --version="false" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073773 4510 flags.go:64] FLAG: --vmodule="" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073777 4510 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.073780 4510 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073936 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073940 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073943 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073947 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073950 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073953 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073956 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073959 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073961 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073964 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073967 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073970 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073973 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.085406 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073976 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073979 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073981 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073984 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073987 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073990 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073992 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073995 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.073999 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074002 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074004 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074007 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074010 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074012 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074015 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074018 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074021 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074024 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074027 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074030 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.086054 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074033 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074035 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074038 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074041 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074043 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074046 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074049 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074052 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074054 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074057 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074059 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074062 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074064 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074067 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074070 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074072 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074075 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074078 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074080 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074083 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.086723 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074087 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074090 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074092 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074095 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074097 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074102 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074106 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074109 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074114 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074117 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074119 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074122 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074125 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074127 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074130 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074132 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074136 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074138 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074141 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.087248 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074143 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074146 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074148 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074151 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074154 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074156 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074159 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074161 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074165 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074169 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074172 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074175 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074178 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074182 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074185 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074188 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074190 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074193 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074195 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.087800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074198 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074200 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074204 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074207 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074210 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074212 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074215 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074218 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074220 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074223 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074226 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074229 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074231 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074234 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074237 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.074263 4510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:54.088311 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074299 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074304 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074306 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074310 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074315 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074318 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074321 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074324 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074327 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074330 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074333 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074336 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074338 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074341 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074344 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074346 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074349 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074351 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074354 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.088918 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074357 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074360 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074362 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074365 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074367 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074370 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074374 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074378 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074380 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074383 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074386 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074389 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074392 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074394 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074397 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074400 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074402 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074405 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074407 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.089422 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074410 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074413 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074415 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074418 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074420 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074423 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074425 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074428 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074431 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074434 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074436 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074439 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074441 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074444 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074446 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074449 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074451 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074454 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074456 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074459 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.089916 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074462 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074464 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074467 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074470 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074473 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074475 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074479 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074482 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074484 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074487 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074490 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074493 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074495 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074498 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074500 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074503 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074506 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074508 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074511 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074513 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.090430 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074528 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074531 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074533 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074536 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074538 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074541 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074544 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074546 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074549 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074551 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074554 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074557 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074559 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074562 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074564 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074567 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074570 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074573 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074575 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074578 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.090958 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074581 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074583 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074587 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074590 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074592 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074594 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074597 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.074599 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.078249 4510 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.078264 4510 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.078283 4510 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.078288 4510 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078344 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078349 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078353 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078357 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078360 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078363 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078366 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.091436 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078369 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078371 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078374 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078377 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078380 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078382 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078385 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078388 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078391 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078393 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078396 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078399 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078402 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078405 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078408 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078411 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078414 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078416 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078419 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078422 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.091924 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078424 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078427 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078430 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078433 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078436 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078438 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078441 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078443 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078446 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078449 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078451 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078453 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078456 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078460 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078464 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078467 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078470 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078473 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078475 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.092397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078478 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078480 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078483 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078485 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078489 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078492 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078494 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078497 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078499 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078502 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078505 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078507 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078510 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078513 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078515 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078530 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078533 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078536 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078554 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078558 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.092901 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078562 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078565 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078568 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078571 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078574 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078578 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078582 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078584 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078587 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078590 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078592 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078595 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078598 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078601 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078604 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078606 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078609 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078612 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078615 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.093383 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078617 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078620 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078622 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078625 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078628 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078630 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078633 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078635 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078638 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078640 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078643 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078646 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078648 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078651 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078653 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078656 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078658 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078661 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078663 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078666 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.093937 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078668 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.078698 4510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078849 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078855 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078858 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078917 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078922 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078925 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078928 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078932 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078936 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078941 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.078989 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079003 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079007 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.094413 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079011 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079016 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079021 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079026 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079032 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079037 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079043 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079048 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079062 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079066 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079071 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079075 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079079 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079085 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079089 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079094 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079100 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079124 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079131 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079138 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.094826 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079145 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079157 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079166 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079181 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079189 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079197 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079204 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079212 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079220 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079227 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079234 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079241 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079248 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079255 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079269 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079277 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079284 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079291 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079298 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079305 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.095301 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079313 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079321 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079329 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079337 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079345 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079352 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079365 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079375 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079382 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079389 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079398 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079404 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079412 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079419 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079426 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079439 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079446 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079454 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079461 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079474 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.095800 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079482 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079489 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079495 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079501 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079508 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079528 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079534 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079541 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079548 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079555 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079561 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079565 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079575 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079579 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079584 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079589 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079594 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079598 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079602 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079609 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.096314 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079614 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079618 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079623 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079632 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079636 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079645 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079650 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079658 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079665 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079671 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079675 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079680 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079684 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.079754 4510 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079874 4510 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:12:54.097008 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079881 4510 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079887 4510 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079891 4510 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079895 4510 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079900 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079910 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079914 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079919 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079923 4510 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079927 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079931 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079936 4510 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079940 4510 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079945 4510 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079949 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079953 4510 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079958 4510 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079963 4510 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079972 4510 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079976 4510 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:12:54.097619 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079981 4510 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079986 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079990 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079995 4510 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.079999 4510 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080005 4510 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080010 4510 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080014 4510 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080019 4510 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080063 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080068 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080072 4510 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080075 4510 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080078 4510 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080081 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080084 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080087 4510 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080090 4510 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080092 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080096 4510 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:12:54.098230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080100 4510 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080117 4510 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080130 4510 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080133 4510 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080136 4510 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080140 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080144 4510 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080148 4510 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080155 4510 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080308 4510 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080331 4510 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080337 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080344 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080349 4510 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080355 4510 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080359 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080365 4510 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080369 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:12:54.098755 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080375 4510 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080380 4510 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080384 4510 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080389 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080395 4510 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080399 4510 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080404 4510 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080409 4510 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080414 4510 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080418 4510 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080427 4510 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080432 4510 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080437 4510 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080441 4510 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080446 4510 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080451 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080455 4510 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080460 4510 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080464 4510 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080469 4510 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:12:54.099214 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080474 4510 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080478 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080482 4510 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080487 4510 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080493 4510 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080497 4510 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080502 4510 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080506 4510 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080511 4510 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080535 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080540 4510 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080545 4510 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080549 4510 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080554 4510 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080558 4510 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080562 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080567 4510 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080571 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080575 4510 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:12:54.099765 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080585 4510 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080589 4510 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080593 4510 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080598 4510 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080603 4510 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080607 4510 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080612 4510 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:54.080619 4510 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.080724 4510 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.082292 4510 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.082372 4510 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.082656 4510 server.go:1014] "Starting client certificate rotation" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.082788 4510 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.082838 4510 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:07:13 +0000 UTC" deadline="2028-08-09 05:48:54.738573196 +0000 UTC" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.082874 4510 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15488h36m0.655701251s" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.083422 4510 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:54.100215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.085107 4510 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:12:54.100823 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.086674 4510 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:12:54.100823 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.088252 4510 log.go:25] "Validated CRI v1 image API" Nov 02 21:12:54.100823 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.088945 4510 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:12:54.100823 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.090495 4510 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9bc9ea3a-b093-4227-ab0d-d1cbc37f76b4:/dev/nvme0n1p3 e6e9f5e6-ab13-4a08-896e-9fabe2c4f7c8:/dev/nvme0n1p4] Nov 02 21:12:54.100942 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.090510 4510 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c71ff9fae033cdba98fdc64ee3320f1a3e88622c9594989f82553aa56f38f93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c71ff9fae033cdba98fdc64ee3320f1a3e88622c9594989f82553aa56f38f93/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29cde747bf78d802fdba65f7ce7ddec8cf02e20ae76cd61f421bf2c3d2619c48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29cde747bf78d802fdba65f7ce7ddec8cf02e20ae76cd61f421bf2c3d2619c48/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32574be367160ba67b510993d900dcc2b64b769b7fd1a309bbd519eaa909cfd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32574be367160ba67b510993d900dcc2b64b769b7fd1a309bbd519eaa909cfd3/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37210afdd2441124ca45894b5478ed520e5c18b8c04b6e9cc1f15cb240c23a1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37210afdd2441124ca45894b5478ed520e5c18b8c04b6e9cc1f15cb240c23a1d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ac38e865a9cbfaed57dac6b82064cc581ad5765292a96c1eedc071ff76fd6dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ac38e865a9cbfaed57dac6b82064cc581ad5765292a96c1eedc071ff76fd6dc/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/498871e6112d2b739fc8bf7de678d7079989ffd9fdaeb34a5d4194786f49f365/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/498871e6112d2b739fc8bf7de678d7079989ffd9fdaeb34a5d4194786f49f365/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/528a47cb352e2868f8f32e37aa9878d8767224397a3e1a0e3764dc0624a81890/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/528a47cb352e2868f8f32e37aa9878d8767224397a3e1a0e3764dc0624a81890/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5594eef07663150f8b75062d60b418657fd5e26f1fbe38b8bea22b16dcb27266/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5594eef07663150f8b75062d60b418657fd5e26f1fbe38b8bea22b16dcb27266/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/779b73cb05a4b93cd8fb22397823417f9c584f92466dd3f6dcfeca5a730aa854/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/779b73cb05a4b93cd8fb22397823417f9c584f92466dd3f6dcfeca5a730aa854/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7975e295f560a9928016070bddf5aeb7e767dbeef3e41a945daadd5bb2b6ccf9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7975e295f560a9928016070bddf5aeb7e767dbeef3e41a945daadd5bb2b6ccf9/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99faea52c621a4233bb363093e1527c278d08bd5944a54df56697d9d3cc6ab8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99faea52c621a4233bb363093e1527c278d08bd5944a54df56697d9d3cc6ab8c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cba58c8d08ee268aa059706241db88ea57b9666ce7dff64166a2b7617f9223f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cba58c8d08ee268aa059706241db88ea57b9666ce7dff64166a2b7617f9223f1/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/faf490672a57df24674a88352d41d036f0f777ad45a1109142796ee59755e0c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/faf490672a57df24674a88352d41d036f0f777ad45a1109142796ee59755e0c2/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f56336c-5149-4436-8bdd-28b2d1b1573f/volumes/kubernetes.io~projected/kube-api-access-k6n9z:{mountpoint:/var/lib/kubelet/pods/0f56336c-5149-4436-8bdd-28b2d1b1573f/volumes/kubernetes.io~projected/kube-api-access-k6n9z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~projected/kube-api-access-xhl65:{mountpoint:/var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~projected/kube-api-access-xhl65 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bfd56dd-b587-42d9-83f8-6788d298b743/volumes/kubernetes.io~projected/kube-api-access-hbrtm:{mountpoint:/var/lib/kubelet/pods/7bfd56dd-b587-42d9-83f8-6788d298b743/volumes/kubernetes.io~projected/kube-api-access-hbrtm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85da7cd6-2547-44ec-a373-e4af9726b87b/volumes/kubernetes.io~projected/kube-api-access-2fj5v:{mountpoint:/var/lib/kubelet/pods/85da7cd6-2547-44ec-a373-e4af9726b87b/volumes/kubernetes.io~projected/kube-api-access-2fj5v major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acb383fe-f128-49fe-b261-c2d7f5d4300f/volumes/kubernetes.io~projected/kube-api-access-rcxv2:{mountpoint:/var/lib/kubelet/pods/acb383fe-f128-49fe-b261-c2d7f5d4300f/volumes/kubernetes.io~projected/kube-api-access-rcxv2 major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acb383fe-f128-49fe-b261-c2d7f5d4300f/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/acb383fe-f128-49fe-b261-c2d7f5d4300f/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b08981c9-b9bb-4f2a-aa0e-30a1be35d954/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b08981c9-b9bb-4f2a-aa0e-30a1be35d954/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1c32e61-e26d-4b32-9a55-4a6770edd655/volumes/kubernetes.io~projected/kube-api-access-26dqm:{mountpoint:/var/lib/kubelet/pods/b1c32e61-e26d-4b32-9a55-4a6770edd655/volumes/kubernetes.io~projected/kube-api-access-26dqm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1c32e61-e26d-4b32-9a55-4a6770edd655/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b1c32e61-e26d-4b32-9a55-4a6770edd655/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8bb07a2-a355-485e-a0f3-b48d5d8f9862/volumes/kubernetes.io~projected/kube-api-access-rnpkz:{mountpoint:/var/lib/kubelet/pods/b8bb07a2-a355-485e-a0f3-b48d5d8f9862/volumes/kubernetes.io~projected/kube-api-access-rnpkz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba030b9f-ec13-45a5-82a9-df4315256051/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ba030b9f-ec13-45a5-82a9-df4315256051/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volumes/kubernetes.io~projected/kube-api-access-59kff:{mountpoint:/var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volumes/kubernetes.io~projected/kube-api-access-59kff major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9/volumes/kubernetes.io~projected/kube-api-access-p944v:{mountpoint:/var/lib/kubelet/pods/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9/volumes/kubernetes.io~projected/kube-api-access-p944v major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce6513b3-2ba0-40fd-b5a3-517dfd17327b/volumes/kubernetes.io~projected/kube-api-access-hmql8:{mountpoint:/var/lib/kubelet/pods/ce6513b3-2ba0-40fd-b5a3-517dfd17327b/volumes/kubernetes.io~projected/kube-api-access-hmql8 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcec323e-b13a-4bc2-b83d-246ada8bbbb3/volumes/kubernetes.io~projected/kube-api-access-kqkbv:{mountpoint:/var/lib/kubelet/pods/dcec323e-b13a-4bc2-b83d-246ada8bbbb3/volumes/kubernetes.io~projected/kube-api-access-kqkbv major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e/volumes/kubernetes.io~projected/kube-api-access-dtstl:{mountpoint:/var/lib/kubelet/pods/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e/volumes/kubernetes.io~projected/kube-api-access-dtstl major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df44d132-0233-4799-9b13-6c5c314fe8c2/volumes/kubernetes.io~projected/kube-api-access-rdjpz:{mountpoint:/var/lib/kubelet/pods/df44d132-0233-4799-9b13-6c5c314fe8c2/volumes/kubernetes.io~projected/kube-api-access-rdjpz major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef5d3dd9-d000-43dc-bf45-fbae72cdd767/volumes/kubernetes.io~projected/kube-api-access-9xtp6:{mountpoint:/var/lib/kubelet/pods/ef5d3dd9-d000-43dc-bf45-fbae72cdd767/volumes/kubernetes.io~projected/kube-api-access-9xtp6 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f955fe-25c7-49e5-8c6b-a9abea532bc1/volumes/kubernetes.io~projected/kube-api-access-9wq67:{mountpoint:/var/lib/kubelet/pods/f0f955fe-25c7-49e5-8c6b-a9abea532bc1/volumes/kubernetes.io~projected/kube-api-access-9wq67 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f84f0d81-e665-4b14-9680-c2dbae153f43/volumes/kubernetes.io~projected/kube-api-access-hjdkb:{mountpoint:/var/lib/kubelet/pods/f84f0d81-e665-4b14-9680-c2dbae153f43/volumes/kubernetes.io~projected/kube-api-access-hjdkb major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/10ac38ccc3d35b1438bd8ae45e3695ef627bc4669f8d1f09bf8d0ad465545317/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ee557514b27f7eb203193d44759c29e436d1fee833d27f84ff4f14c23ccb5c20/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/38e1ced847eda170757be9b60f692576966d5ef1cdd302034f5cfe3dad64fdab/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/0be8695bd23c8dba837857521bd36a4b427aa70851a03738b162af2c6caa864c/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/71faa6e8827e30685c432627cd6b04962f15cf84dde1922269f88dcdca63899d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5668776acc31edbc4cd8d030601c6fce5ed516155d9e2a08d147de1f266b4fbf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9b4da1b79ae3a557e3b8cbce1195c17d96d44e9dcaee280ff2c25a92afb4a72a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0263d8ffd86568e22b2bde39b412cad2560bf65522527ffe128193d32f0de253/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2f166329dde5356ffa3899642a0cb56483e4c6b56621b48055518a12779be4a8/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/4a67286274763d2b5e580810542c74f456c0cf824770b3ef7dab379331da2bd3/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/3113b7c3a5f1150e076e43a8a3361908f36459aecd59dc0827e66bf98732d4ad/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/2b89ee8dc19a8881eb23457982818104f6f21c8390dce0a05acb5d89f43bf29d/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/42247756eba6cd877f0bd05b5bd31fdca61e9ca3b182cacbaea94a98f75b84a0/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/767640ebcf887cde5a8387169bf2d246490376ab993376731879832872a2db0f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e9ef307a1a298909ca7eb96231987bf57671f4fcd221d592e6a3bd0eea41f7e9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3269a011ce8862a66438948b7efda064eb2e3715f27573b0bbcbb228fce3f002/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6aefb913743ea6d7ab2b34bd937a6420b0cfa2f504418899ef1d7b0dc3c977e4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ab4ba499f18ed5b169fea8b53062a5d2f8c6f6fca048104e75e1567abae45117/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2b49cd6b301a91f5e84060782eba62d60eef2e52ac3dee7269d175c967bca00a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/292a1f55b2a49794930606e034cc60ad1c9cad1934c0b2c3cb132b86b36054ad/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4fd87e5ee4ac9629db9c2987e84475d77912b439f412f48b04ce68c4d034117b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/37b9fc4546df0cebf262d4faebb0c521b0c92d4505c96e87878e24942a7716f6/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/88b9c61f35986ed7d122f4802851848f0e3ff8c4b719cc3539cf169db9676818/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/7186ed3e993c97d986ff26e7228c642ec067d5587840675bbf81dd4219a9b60b/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/16fbfb0cf836d7fed31783e28d2e61ed1c431b542de715569a3b67b5b3deb3a4/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/c7bc5fa1f3a48d993715294916774c75ecc4d4faa6111d30c177d9180f523eee/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/159b54579e6a9446f9bdab4aff4c18b82cdda490fc4d868113e9e4033dde1a0e/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/822a18ff9ffe94dd00102d62d5ace0eee5bcfa01cbedec344bb2470e287d2d5f/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/f7189fa14c71d3ec414323f8d560aae46c1ca3502efbcf83e3dd205ea3e580e2/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/9076ef3399cd6df09bef3a0ef2010a31ce7cad6a8b1d4959845bf9f8b1945535/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/4d0a0beca9b15ac9b3a84db7234e6469cfa2a31b26e0083b34d61d059b649991/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/d8895f3fc3bdbd32286b3e97d7b9d17113a7d3d788b1fadc71d51dda85329919/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/177cfdb1e711ef25842bb71bc1ffde205734b3aca9275c21e342701b54a87bb0/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f00ad382f0450e9f9cef4c999473856598ae39e8cc695ec2876d9ceaeeae8e18/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e1fe7f31b5e0b43479ef21d7095ac16c5836bf3b4ead8874b440c158c5ebba91/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7a15e42b8d263c3fffeb708303430747b2eef72a53a6964146d302848b52904b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/391638a6289d1b9be54ffae4641928428cb086949935b38bcbac1214a72f9ea9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/d28cb23cd7d1700d46b75804480fde5d4bc0981744b421dff4bc2fa3fd381a54/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ec9f20a2bda952f58ea3b1f15fe87065472e35e4e6a8349104b02f92618132f3/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f2053d3184f4daeeeb59a5c121f9e3e2152e79eba62e369c5bc0ed639e8138d4/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a1f58b32c913b580049b2f7005e67eb58b1254e9b2ddf65d2a73657eaf12b647/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/265592d7fa324f715f2abdb36156a1eb883d86cede1149fca9a504e3e8ffe023/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098012 4510 manager.go:217] Machine: {Timestamp:2026-11-02 21:12:54.097035597 +0000 UTC m=+0.073333446 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127545 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21df34985c13c3e9400ca9794e2bf5 SystemUUID:ec21df34-985c-13c3-e940-0ca9794e2bf5 BootID:d7709a17-3d94-4e72-8828-fda4e2c844b6 Filesystems:[{Device:/run/containers/storage/overlay-containers/79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c71ff9fae033cdba98fdc64ee3320f1a3e88622c9594989f82553aa56f38f93/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5594eef07663150f8b75062d60b418657fd5e26f1fbe38b8bea22b16dcb27266/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1c32e61-e26d-4b32-9a55-4a6770edd655/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9/volumes/kubernetes.io~projected/kube-api-access-p944v DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/528a47cb352e2868f8f32e37aa9878d8767224397a3e1a0e3764dc0624a81890/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/779b73cb05a4b93cd8fb22397823417f9c584f92466dd3f6dcfeca5a730aa854/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba030b9f-ec13-45a5-82a9-df4315256051/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/faf490672a57df24674a88352d41d036f0f777ad45a1109142796ee59755e0c2/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b8bb07a2-a355-485e-a0f3-b48d5d8f9862/volumes/kubernetes.io~projected/kube-api-access-rnpkz DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f84f0d81-e665-4b14-9680-c2dbae153f43/volumes/kubernetes.io~projected/kube-api-access-hjdkb DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/498871e6112d2b739fc8bf7de678d7079989ffd9fdaeb34a5d4194786f49f365/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce6513b3-2ba0-40fd-b5a3-517dfd17327b/volumes/kubernetes.io~projected/kube-api-access-hmql8 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acb383fe-f128-49fe-b261-c2d7f5d4300f/volumes/kubernetes.io~projected/kube-api-access-rcxv2 DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7bfd56dd-b587-42d9-83f8-6788d298b743/volumes/kubernetes.io~projected/kube-api-access-hbrtm DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f955fe-25c7-49e5-8c6b-a9abea532bc1/volumes/kubernetes.io~projected/kube-api-access-9wq67 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99faea52c621a4233bb363093e1527c278d08bd5944a54df56697d9d3cc6ab8c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32574be367160ba67b510993d900dcc2b64b769b7fd1a309bbd519eaa909cfd3/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~projected/kube-api-access-xhl65 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e/volumes/kubernetes.io~projected/kube-api-access-dtstl DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ac38e865a9cbfaed57dac6b82064cc581ad5765292a96c1eedc071ff76fd6dc/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/0f56336c-5149-4436-8bdd-28b2d1b1573f/volumes/kubernetes.io~projected/kube-api-access-k6n9z DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c66fda8b-8c6a-480a-9f42-656b485bc49a/volumes/kubernetes.io~projected/kube-api-access-59kff DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cba58c8d08ee268aa059706241db88ea57b9666ce7dff64166a2b7617f9223f1/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29cde747bf78d802fdba65f7ce7ddec8cf02e20ae76cd61f421bf2c3d2619c48/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df44d132-0233-4799-9b13-6c5c314fe8c2/volumes/kubernetes.io~projected/kube-api-access-rdjpz DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b08981c9-b9bb-4f2a-aa0e-30a1be35d954/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acb383fe-f128-49fe-b261-c2d7f5d4300f/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ac06bad-5657-47d1-9f15-87fd58f9945f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef5d3dd9-d000-43dc-bf45-fbae72cdd767/volumes/kubernetes.io~projected/kube-api-access-9xtp6 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85da7cd6-2547-44ec-a373-e4af9726b87b/volumes/kubernetes.io~projected/kube-api-access-2fj5v DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dcec323e-b13a-4bc2-b83d-246ada8bbbb3/volumes/kubernetes.io~projected/kube-api-access-kqkbv DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7975e295f560a9928016070bddf5aeb7e767dbeef3e41a945daadd5bb2b6ccf9/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1c32e61-e26d-4b32-9a55-4a6770edd655/volumes/kubernetes.io~projected/kube-api-access-26dqm DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37210afdd2441124ca45894b5478ed520e5c18b8c04b6e9cc1f15cb240c23a1d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1c71ff9fae033cd MacAddress:5e:da:9d:6f:37:89 Speed:10000 Mtu:8901} {Name:32574be367160ba MacAddress:06:bc:72:98:d0:0c Speed:10000 Mtu:8901} {Name:37210afdd244112 MacAddress:2a:45:cf:e0:41:11 Speed:10000 Mtu:8901} {Name:b24d932dcb79552 MacAddress:c2:76:f9:b7:07:23 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e7:5a:2d:b5 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:96:a9:3f:6d:49 Speed:0 Mtu:8901} {Name:cba58c8d08ee268 MacAddress:e2:51:56:56:1b:04 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e7:5a:2d:b5 Speed:0 Mtu:9001} {Name:faf490672a57df2 MacAddress:6e:ee:bd:d8:f5:52 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:9e:b6:86:1e:ed:f3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:39:e9:fa:2c:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098353 4510 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098405 4510 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098684 4510 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098719 4510 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-33.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098847 4510 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098854 4510 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098864 4510 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098883 4510 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.098967 4510 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.099130 4510 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.099240 4510 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.099275 4510 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.099290 4510 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.099300 4510 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.099309 4510 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.099316 4510 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.100104 4510 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.100117 4510 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:54.101079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.100669 4510 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:12:54.111372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111356 4510 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:12:54.111372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111371 4510 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:12:54.111489 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111475 4510 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 21:12:54.111686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111675 4510 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:12:54.111725 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111704 4510 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:12:54.111798 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111789 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:12:54.111829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111803 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:12:54.111829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111812 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:12:54.111829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111821 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:12:54.111829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111827 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111834 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111839 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111844 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111853 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111859 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111870 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111883 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111909 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:12:54.111937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.111918 4510 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:12:54.115270 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.115256 4510 server.go:1263] "Started kubelet" Nov 02 21:12:54.115791 ip-10-0-133-33 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:12:54.116182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.116013 4510 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:12:54.117362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.115355 4510 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:12:54.117445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.117389 4510 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:12:54.117496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.117448 4510 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:12:54.117662 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.117646 4510 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.118286 4510 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.118416 4510 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.118645 4510 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:07:15 +0000 UTC" deadline="2028-08-01 22:16:55.121690854 +0000 UTC" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.118697 4510 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15313h4m1.00301446s" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.118763 4510 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.118777 4510 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.118954 4510 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.119264 4510 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120846 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acb383fe-f128-49fe-b261-c2d7f5d4300f" volumeName="kubernetes.io/projected/acb383fe-f128-49fe-b261-c2d7f5d4300f-kube-api-access-rcxv2" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120886 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b08981c9-b9bb-4f2a-aa0e-30a1be35d954" volumeName="kubernetes.io/configmap/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-konnectivity-ca" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120902 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66fda8b-8c6a-480a-9f42-656b485bc49a" volumeName="kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-script-lib" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120916 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66fda8b-8c6a-480a-9f42-656b485bc49a" volumeName="kubernetes.io/secret/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120933 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df44d132-0233-4799-9b13-6c5c314fe8c2" volumeName="kubernetes.io/projected/df44d132-0233-4799-9b13-6c5c314fe8c2-kube-api-access-rdjpz" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120945 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b08981c9-b9bb-4f2a-aa0e-30a1be35d954" volumeName="kubernetes.io/secret/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-agent-certs" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120962 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9" volumeName="kubernetes.io/projected/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-kube-api-access-p944v" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.120988 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd4b2790-68ec-4dd1-8085-1aeb5fe3521e" volumeName="kubernetes.io/projected/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kube-api-access-dtstl" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121009 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" volumeName="kubernetes.io/projected/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-kube-api-access-9wq67" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121021 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac06bad-5657-47d1-9f15-87fd58f9945f" volumeName="kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-tuned" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121037 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acb383fe-f128-49fe-b261-c2d7f5d4300f" volumeName="kubernetes.io/secret/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-key" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121054 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66fda8b-8c6a-480a-9f42-656b485bc49a" volumeName="kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-config" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121067 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" volumeName="kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121084 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84f0d81-e665-4b14-9680-c2dbae153f43" volumeName="kubernetes.io/projected/f84f0d81-e665-4b14-9680-c2dbae153f43-kube-api-access-hjdkb" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121100 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f56336c-5149-4436-8bdd-28b2d1b1573f" volumeName="kubernetes.io/projected/0f56336c-5149-4436-8bdd-28b2d1b1573f-kube-api-access-k6n9z" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121117 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac06bad-5657-47d1-9f15-87fd58f9945f" volumeName="kubernetes.io/projected/5ac06bad-5657-47d1-9f15-87fd58f9945f-kube-api-access-xhl65" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121130 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85da7cd6-2547-44ec-a373-e4af9726b87b" volumeName="kubernetes.io/projected/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-api-access-2fj5v" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121146 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1c32e61-e26d-4b32-9a55-4a6770edd655" volumeName="kubernetes.io/projected/b1c32e61-e26d-4b32-9a55-4a6770edd655-kube-api-access-26dqm" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121157 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66fda8b-8c6a-480a-9f42-656b485bc49a" volumeName="kubernetes.io/projected/c66fda8b-8c6a-480a-9f42-656b485bc49a-kube-api-access-59kff" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121168 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df44d132-0233-4799-9b13-6c5c314fe8c2" volumeName="kubernetes.io/configmap/df44d132-0233-4799-9b13-6c5c314fe8c2-config-volume" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121183 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df44d132-0233-4799-9b13-6c5c314fe8c2" volumeName="kubernetes.io/empty-dir/df44d132-0233-4799-9b13-6c5c314fe8c2-tmp-dir" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121288 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef5d3dd9-d000-43dc-bf45-fbae72cdd767" volumeName="kubernetes.io/projected/ef5d3dd9-d000-43dc-bf45-fbae72cdd767-kube-api-access-9xtp6" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121326 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85da7cd6-2547-44ec-a373-e4af9726b87b" volumeName="kubernetes.io/empty-dir/85da7cd6-2547-44ec-a373-e4af9726b87b-data-volume" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121346 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce6513b3-2ba0-40fd-b5a3-517dfd17327b" volumeName="kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121369 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" volumeName="kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121386 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" volumeName="kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121405 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" volumeName="kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-binary-copy" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121420 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f56336c-5149-4436-8bdd-28b2d1b1573f" volumeName="kubernetes.io/empty-dir/0f56336c-5149-4436-8bdd-28b2d1b1573f-tmp-dir" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121604 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bfd56dd-b587-42d9-83f8-6788d298b743" volumeName="kubernetes.io/projected/7bfd56dd-b587-42d9-83f8-6788d298b743-kube-api-access-hbrtm" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121624 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acb383fe-f128-49fe-b261-c2d7f5d4300f" volumeName="kubernetes.io/configmap/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-cabundle" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121646 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1c32e61-e26d-4b32-9a55-4a6770edd655" volumeName="kubernetes.io/secret/b1c32e61-e26d-4b32-9a55-4a6770edd655-cert" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121662 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba030b9f-ec13-45a5-82a9-df4315256051" volumeName="kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121681 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66fda8b-8c6a-480a-9f42-656b485bc49a" volumeName="kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-env-overrides" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121701 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" volumeName="kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121715 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" volumeName="kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121752 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bfd56dd-b587-42d9-83f8-6788d298b743" volumeName="kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-cni-binary-copy" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121777 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8bb07a2-a355-485e-a0f3-b48d5d8f9862" volumeName="kubernetes.io/configmap/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-iptables-alerter-script" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121792 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" volumeName="kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121805 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ac06bad-5657-47d1-9f15-87fd58f9945f" volumeName="kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-tmp" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121826 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bfd56dd-b587-42d9-83f8-6788d298b743" volumeName="kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-daemon-config" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121838 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8bb07a2-a355-485e-a0f3-b48d5d8f9862" volumeName="kubernetes.io/projected/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-kube-api-access-rnpkz" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121846 4510 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84f0d81-e665-4b14-9680-c2dbae153f43" volumeName="kubernetes.io/configmap/f84f0d81-e665-4b14-9680-c2dbae153f43-serviceca" seLinuxMountContext="" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121855 4510 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:12:54.122783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.121860 4510 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.123726 4510 factory.go:55] Registering systemd factory Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.123745 4510 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.123947 4510 factory.go:153] Registering CRI-O factory Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.123961 4510 factory.go:223] Registration of the crio container factory successfully Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.124013 4510 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.124033 4510 factory.go:103] Registering Raw factory Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.124047 4510 manager.go:1199] Started watching for new ooms in manager Nov 02 21:12:54.125432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.124358 4510 manager.go:319] Starting recovery of all containers Nov 02 21:12:54.144427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.144384 4510 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:12:54.145604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.145579 4510 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:12:54.145604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.145606 4510 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:12:54.145769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.145625 4510 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:12:54.145769 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.145668 4510 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 21:12:54.170861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.170831 4510 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="78870204121b5651960a80803a4f947139443462776aa99adc2248714ab46551" exitCode=0 Nov 02 21:12:54.170861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.170861 4510 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="47a2e5392ee5d155e022b8e28bd9e470f9f7ed482374020bdcbe3dd4661ac174" exitCode=0 Nov 02 21:12:54.171025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.170870 4510 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="aab8a764a9395a913b6a7a47475bfa5c525fd0b737784aca3f293dca6d720009" exitCode=0 Nov 02 21:12:54.171025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.170878 4510 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="b5a2137faf117dec91c27d32843591784746967aa37d0316e3b6ccc3b6474f7a" exitCode=0 Nov 02 21:12:54.171025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.170884 4510 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="d9ce078ff5d71798c0fac426c79ea5c4c631079fb50c30f62252781671412d00" exitCode=0 Nov 02 21:12:54.171025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.170892 4510 generic.go:356] "Generic (PLEG): container finished" podID="f0f955fe-25c7-49e5-8c6b-a9abea532bc1" containerID="397fb892b7df4149935fb1b8e29260ee2528acd911377ab4fe7772a1490002d5" exitCode=0 Nov 02 21:12:54.176876 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.176855 4510 generic.go:356] "Generic (PLEG): container finished" podID="c04c921cae53f031e25193b5cbbab217" containerID="a4968370a0eaa7963000d4642cf91324112b294cd790440f7d3e2d9ec7bfe247" exitCode=0 Nov 02 21:12:54.194347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.194288 4510 manager.go:324] Recovery completed Nov 02 21:12:54.208873 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.208853 4510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:54.209951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.209929 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:54.210038 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.209969 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:54.210038 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.209982 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:54.211397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211384 4510 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:12:54.211397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211394 4510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:54.211488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211396 4510 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:12:54.211488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211470 4510 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:12:54.211644 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211629 4510 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 21:12:54.211679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211644 4510 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 21:12:54.211679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211660 4510 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 21:12:54.211679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211666 4510 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 21:12:54.211679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211671 4510 policy_none.go:50] "Start" Nov 02 21:12:54.211787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211683 4510 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:12:54.211787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211693 4510 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:12:54.211851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211841 4510 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 21:12:54.211883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211851 4510 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 21:12:54.211883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.211857 4510 policy_none.go:44] "Start" Nov 02 21:12:54.212235 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.212219 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:54.212297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.212248 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:54.212297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.212262 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:54.215876 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.215854 4510 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:12:54.218826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.218812 4510 manager.go:341] "Starting Device Plugin manager" Nov 02 21:12:54.218915 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.218837 4510 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:12:54.218915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.218847 4510 server.go:87] "Starting device plugin registration server" Nov 02 21:12:54.219108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.219097 4510 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:12:54.219154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.219109 4510 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:12:54.219226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.219208 4510 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:12:54.219357 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.219316 4510 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:12:54.219467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.219361 4510 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:12:54.219467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.219368 4510 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:12:54.219687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.219436 4510 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:12:54.219351635Z","UUID":"e0ff4a8f-4c76-490a-a0b8-52d769f3db5e","Handler":null,"Name":"","Endpoint":""} Nov 02 21:12:54.220511 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.220494 4510 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:12:54.220643 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.220619 4510 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" Nov 02 21:12:54.220840 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.220824 4510 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:12:54.220902 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.220848 4510 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:12:54.246054 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.246015 4510 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal"] Nov 02 21:12:54.246176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.246162 4510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:54.246927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.246910 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:54.247008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.246941 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:54.247008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.246956 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:54.247263 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.247249 4510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:54.247432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.247418 4510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:54.247993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.247972 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:54.248084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248004 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:54.248084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248019 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:54.248084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248026 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:54.248084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248043 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:54.248084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248055 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:54.248240 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.248180 4510 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.248340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248314 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"7a52ae9f924fc84a5af3aafcc66e569a","Type":"ContainerStarted","Data":"b4d10830160ddd7a7c8200f197935d03752459254f35fe7120ca469687be9b8b"} Nov 02 21:12:54.248370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248353 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"7a52ae9f924fc84a5af3aafcc66e569a","Type":"ContainerStarted","Data":"79a34a46723e604c3aa458ab419eaaece33d9ccafc855c812fe13db5bf401b1b"} Nov 02 21:12:54.248421 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248409 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"c04c921cae53f031e25193b5cbbab217","Type":"ContainerStarted","Data":"652033d98dad9fb19575b46e22586a6f85f139d36408ab988acbc4295aec7776"} Nov 02 21:12:54.248454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248427 4510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:54.248454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248436 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"c04c921cae53f031e25193b5cbbab217","Type":"ContainerDied","Data":"a4968370a0eaa7963000d4642cf91324112b294cd790440f7d3e2d9ec7bfe247"} Nov 02 21:12:54.248533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.248456 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" event={"ID":"c04c921cae53f031e25193b5cbbab217","Type":"ContainerStarted","Data":"a3929300ebfe33d0da794c19ce634045c702d32f6105540bfc2e6075096e0a26"} Nov 02 21:12:54.249091 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.249076 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:54.249137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.249100 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:54.249137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.249116 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:54.249249 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.249238 4510 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-33.us-west-2.compute.internal\" not found" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.293812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.293771 4510 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:54.293923 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.293886 4510 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 21:12:54.293923 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.293888 4510 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:54.294072 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.294059 4510 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:54.320148 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.320125 4510 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:12:54.320148 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.320143 4510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.321055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.321040 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:12:54.321119 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.321067 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:12:54.321119 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.321077 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:12:54.321119 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.321112 4510 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.340653 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.340630 4510 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:12:54.385624 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.385598 4510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:54.385755 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.385646 4510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.385755 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.385660 4510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.394899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.394877 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.394977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.394904 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.394977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.394922 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a52ae9f924fc84a5af3aafcc66e569a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"7a52ae9f924fc84a5af3aafcc66e569a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.424944 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.424920 4510 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.425162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.425145 4510 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.425980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.425959 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeNotReady" Nov 02 21:12:54.426083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.425994 4510 setters.go:546] "Node became not ready" node="ip-10-0-133-33.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T21:12:54Z","lastTransitionTime":"2026-11-02T21:12:54Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 21:12:54.439924 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.439898 4510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:54.440019 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:54.439959 4510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.443186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.443170 4510 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-33.us-west-2.compute.internal" is owned by different node (oldNodeID="a94cf43d-2f98-429b-a453-44b238bc6594", newNodeID=""), cleaning up... Nov 02 21:12:54.495903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.495851 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.495903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.495878 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.495903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.495895 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a52ae9f924fc84a5af3aafcc66e569a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"7a52ae9f924fc84a5af3aafcc66e569a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.496047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.495937 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a52ae9f924fc84a5af3aafcc66e569a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"7a52ae9f924fc84a5af3aafcc66e569a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.496047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.495951 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.496047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.495956 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c04c921cae53f031e25193b5cbbab217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" (UID: \"c04c921cae53f031e25193b5cbbab217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:54.690329 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.690295 4510 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-33.us-west-2.compute.internal" event="NodeReady" Nov 02 21:12:54.690455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:54.690375 4510 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:12:55.099541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.099500 4510 apiserver.go:52] "Watching apiserver" Nov 02 21:12:55.187117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.187091 4510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:55.187244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.187226 4510 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:55.197685 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.197662 4510 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:12:55.198141 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.198122 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7","openshift-network-diagnostics/network-check-target-fmkt5","kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal","openshift-insights/insights-runtime-extractor-4c9pl","openshift-multus/multus-ccljr","kube-system/global-pull-secret-syncer-fdpbr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562","openshift-dns/node-resolver-pdftq","openshift-insights/periodic-gathering-szkkx-zrfdx","openshift-ingress-canary/ingress-canary-tmmm8","openshift-network-operator/iptables-alerter-fbktz","kube-system/konnectivity-agent-wcgz4","openshift-dns/dns-default-rrj8p","openshift-service-ca/service-ca-5fffc97bdf-s59nj","openshift-ovn-kubernetes/ovnkube-node-p7wfq","openshift-multus/multus-additional-cni-plugins-qhkj4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal","openshift-image-registry/node-ca-f6955","openshift-cluster-node-tuning-operator/tuned-q4ljn","openshift-multus/network-metrics-daemon-b26fp"] Nov 02 21:12:55.198961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.198933 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:55.202577 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.202555 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.204161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.204144 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.219974 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.219952 4510 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:12:55.243451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.243419 4510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:55.243583 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.243469 4510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:55.243745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.243728 4510 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:12:55.243810 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.243780 4510 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-33.us-west-2.compute.internal" Nov 02 21:12:55.250405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.250382 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:12:55.250567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.250402 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:12:55.250567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.250403 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.250722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.250440 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.250859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.250843 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.250919 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.250907 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.251025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251012 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:12:55.251144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251126 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:12:55.251220 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251131 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:12:55.251220 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251193 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.251220 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251208 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c6fz8\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251250 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nvsgt\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251259 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mnft8\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251279 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251291 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c5zxb\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251305 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251308 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z2vtc\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251326 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251332 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251336 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dtndn\"" Nov 02 21:12:55.251362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251343 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251370 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251377 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251346 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251391 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ps8jp\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251402 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sdlzm\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251411 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251380 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251396 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-q4vfp\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251419 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251441 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-r4znh\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251427 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.251733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.251476 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:12:55.283340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283314 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:12:55.283492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283426 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5dfd2\"" Nov 02 21:12:55.283492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283427 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tjjtb\"" Nov 02 21:12:55.283492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283488 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zzckx\"" Nov 02 21:12:55.283777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283757 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:12:55.283846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283792 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.283846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283803 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 21:12:55.283942 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283845 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 21:12:55.284008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.283991 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:12:55.284056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.284045 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lplw4\"" Nov 02 21:12:55.290002 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.289986 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.290394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.290379 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:12:55.290394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.290388 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhk5v\"" Nov 02 21:12:55.290486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.290400 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.295974 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.295957 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.296056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.295974 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:12:55.296056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.295980 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:12:55.296056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296000 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:12:55.296056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296008 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:12:55.296056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296034 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:12:55.296056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296045 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-crvzm\"" Nov 02 21:12:55.296848 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296835 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 21:12:55.296909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296868 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.296952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296911 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w8spg\"" Nov 02 21:12:55.297004 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.296968 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.297137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297122 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jzn4w\"" Nov 02 21:12:55.297352 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297327 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.297419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297344 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 21:12:55.297747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297723 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:12:55.297812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297737 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.297812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297778 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:12:55.297887 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297809 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:12:55.297966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.297945 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 21:12:55.299801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299784 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-systemd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.299918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299818 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-var-lib-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.299918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299843 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-bin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.299918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299894 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-bin\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299926 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299955 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299973 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kff\" (UniqueName: \"kubernetes.io/projected/c66fda8b-8c6a-480a-9f42-656b485bc49a-kube-api-access-59kff\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.299988 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.300019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300012 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c32e61-e26d-4b32-9a55-4a6770edd655-cert\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:55.300170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300027 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.300170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300041 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-var-lib-kubelet\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.300170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300075 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cnibin\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.300170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300101 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjpz\" (UniqueName: \"kubernetes.io/projected/df44d132-0233-4799-9b13-6c5c314fe8c2-kube-api-access-rdjpz\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.300170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300135 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-node-log\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300193 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovn-node-metrics-cert\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300256 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300296 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.300325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300306 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c32e61-e26d-4b32-9a55-4a6770edd655-cert\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:55.300468 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300346 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.300468 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300386 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-cabundle\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:55.300468 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300416 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-netns\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.300468 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300433 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrtm\" (UniqueName: \"kubernetes.io/projected/7bfd56dd-b587-42d9-83f8-6788d298b743-kube-api-access-hbrtm\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.300647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300494 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-etc-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300555 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-log-socket\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300621 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-netd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.300647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300636 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-cabundle\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:55.300769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300651 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.300769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300667 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysconfig\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.300769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300677 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-binary-copy\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.300769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300684 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da7cd6-2547-44ec-a373-e4af9726b87b-crio-socket\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.300769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300726 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-socket-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.300769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300759 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.300930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300762 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f56336c-5149-4436-8bdd-28b2d1b1573f-hosts-file\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:55.300930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300807 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:55.300930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300826 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhl65\" (UniqueName: \"kubernetes.io/projected/5ac06bad-5657-47d1-9f15-87fd58f9945f-kube-api-access-xhl65\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.300930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300896 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtstl\" (UniqueName: \"kubernetes.io/projected/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kube-api-access-dtstl\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.301036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.300977 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6n9z\" (UniqueName: \"kubernetes.io/projected/0f56336c-5149-4436-8bdd-28b2d1b1573f-kube-api-access-k6n9z\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:55.301036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301010 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-conf\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.301088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301039 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.301164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301149 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.301198 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301181 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-kubelet\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.301232 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301204 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-kubelet\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.301261 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301227 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.301261 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301246 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.301325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301277 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-modprobe-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.301325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301309 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-sys\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.301387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301328 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-hostroot\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.301387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301370 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-tuned\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.301447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301422 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-tuned\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.301482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301443 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-iptables-alerter-script\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:55.301482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301469 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpkz\" (UniqueName: \"kubernetes.io/projected/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-kube-api-access-rnpkz\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:55.301560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301505 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-device-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.301592 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301569 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df44d132-0233-4799-9b13-6c5c314fe8c2-tmp-dir\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.301632 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301594 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkbv\" (UniqueName: \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.301632 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301619 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-key\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:55.301715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301643 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-cni-binary-copy\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.301775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301756 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df44d132-0233-4799-9b13-6c5c314fe8c2-tmp-dir\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.301831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301809 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-kubelet-config\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.301896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301863 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdkb\" (UniqueName: \"kubernetes.io/projected/f84f0d81-e665-4b14-9680-c2dbae153f43-kube-api-access-hjdkb\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:55.301952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301892 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-config\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.301952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301898 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-iptables-alerter-script\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:55.301952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301917 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dqm\" (UniqueName: \"kubernetes.io/projected/b1c32e61-e26d-4b32-9a55-4a6770edd655-kube-api-access-26dqm\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:55.301952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301943 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.302098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301970 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcxv2\" (UniqueName: \"kubernetes.io/projected/acb383fe-f128-49fe-b261-c2d7f5d4300f-kube-api-access-rcxv2\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:55.302098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301974 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-cni-binary-copy\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.302098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.301995 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-system-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.302098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302009 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/acb383fe-f128-49fe-b261-c2d7f5d4300f-signing-key\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:55.302098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302019 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-multus\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.302098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302064 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-ovn\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.302339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302195 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fj5v\" (UniqueName: \"kubernetes.io/projected/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-api-access-2fj5v\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.302339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302216 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-systemd\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.302339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302234 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.302339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302250 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.302339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302309 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.302339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302308 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.302339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302337 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-env-overrides\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302363 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-lib-modules\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302404 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-multus-certs\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302440 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-system-cni-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302482 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-config\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302512 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302573 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-conf-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302584 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-env-overrides\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302623 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-script-lib\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302656 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302683 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-host-slash\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:55.302686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302690 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302720 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da7cd6-2547-44ec-a373-e4af9726b87b-data-volume\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302748 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba030b9f-ec13-45a5-82a9-df4315256051-original-pull-secret\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302768 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-registration-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302810 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-sys-fs\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302821 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85da7cd6-2547-44ec-a373-e4af9726b87b-data-volume\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302839 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p944v\" (UniqueName: \"kubernetes.io/projected/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-kube-api-access-p944v\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302846 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66fda8b-8c6a-480a-9f42-656b485bc49a-ovnkube-script-lib\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302868 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d132-0233-4799-9b13-6c5c314fe8c2-config-volume\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.302910 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-cnibin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303022 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f84f0d81-e665-4b14-9680-c2dbae153f43-serviceca\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303040 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-systemd-units\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303060 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtp6\" (UniqueName: \"kubernetes.io/projected/ef5d3dd9-d000-43dc-bf45-fbae72cdd767-kube-api-access-9xtp6\") pod \"migrator-66df5d496b-tfnj7\" (UID: \"ef5d3dd9-d000-43dc-bf45-fbae72cdd767\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303084 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-tmp\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303084 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df44d132-0233-4799-9b13-6c5c314fe8c2-config-volume\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303104 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.303154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303159 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wq67\" (UniqueName: \"kubernetes.io/projected/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-kube-api-access-9wq67\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303210 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ac06bad-5657-47d1-9f15-87fd58f9945f-tmp\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303231 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f84f0d81-e665-4b14-9680-c2dbae153f43-serviceca\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303251 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303287 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-socket-dir-parent\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303316 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f84f0d81-e665-4b14-9680-c2dbae153f43-host\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303351 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-netns\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303383 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303409 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303446 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-kubernetes\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303475 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-run\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303509 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-konnectivity-ca\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303582 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-k8s-cni-cncf-io\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303626 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-daemon-config\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303691 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f56336c-5149-4436-8bdd-28b2d1b1573f-tmp-dir\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303730 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-host\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303732 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-konnectivity-ca\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303751 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f56336c-5149-4436-8bdd-28b2d1b1573f-tmp-dir\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303756 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-agent-certs\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303786 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-etc-kubernetes\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303822 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-daemon-config\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303846 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-dbus\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.303838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.303865 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-slash\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.304458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.304004 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b08981c9-b9bb-4f2a-aa0e-30a1be35d954-agent-certs\") pod \"konnectivity-agent-wcgz4\" (UID: \"b08981c9-b9bb-4f2a-aa0e-30a1be35d954\") " pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:12:55.352691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.352619 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrtm\" (UniqueName: \"kubernetes.io/projected/7bfd56dd-b587-42d9-83f8-6788d298b743-kube-api-access-hbrtm\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.352691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.352662 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kff\" (UniqueName: \"kubernetes.io/projected/c66fda8b-8c6a-480a-9f42-656b485bc49a-kube-api-access-59kff\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.352691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.352629 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjpz\" (UniqueName: \"kubernetes.io/projected/df44d132-0233-4799-9b13-6c5c314fe8c2-kube-api-access-rdjpz\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.355695 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.355673 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpkz\" (UniqueName: \"kubernetes.io/projected/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-kube-api-access-rnpkz\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:55.355833 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.355815 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdkb\" (UniqueName: \"kubernetes.io/projected/f84f0d81-e665-4b14-9680-c2dbae153f43-kube-api-access-hjdkb\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:55.356055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.356037 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtstl\" (UniqueName: \"kubernetes.io/projected/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kube-api-access-dtstl\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.356843 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.356149 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6n9z\" (UniqueName: \"kubernetes.io/projected/0f56336c-5149-4436-8bdd-28b2d1b1573f-kube-api-access-k6n9z\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:55.356926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.356911 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkbv\" (UniqueName: \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\") pod \"periodic-gathering-szkkx-zrfdx\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:12:55.357258 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.357199 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhl65\" (UniqueName: \"kubernetes.io/projected/5ac06bad-5657-47d1-9f15-87fd58f9945f-kube-api-access-xhl65\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.357951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.357931 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtp6\" (UniqueName: \"kubernetes.io/projected/ef5d3dd9-d000-43dc-bf45-fbae72cdd767-kube-api-access-9xtp6\") pod \"migrator-66df5d496b-tfnj7\" (UID: \"ef5d3dd9-d000-43dc-bf45-fbae72cdd767\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tfnj7" Nov 02 21:12:55.361100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.361080 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p944v\" (UniqueName: \"kubernetes.io/projected/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-kube-api-access-p944v\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:55.361256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.361233 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fj5v\" (UniqueName: \"kubernetes.io/projected/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-api-access-2fj5v\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.361314 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.361271 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dqm\" (UniqueName: \"kubernetes.io/projected/b1c32e61-e26d-4b32-9a55-4a6770edd655-kube-api-access-26dqm\") pod \"ingress-canary-tmmm8\" (UID: \"b1c32e61-e26d-4b32-9a55-4a6770edd655\") " pod="openshift-ingress-canary/ingress-canary-tmmm8" Nov 02 21:12:55.361388 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.361365 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wq67\" (UniqueName: \"kubernetes.io/projected/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-kube-api-access-9wq67\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.361424 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.361408 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmql8\" (UniqueName: \"kubernetes.io/projected/ce6513b3-2ba0-40fd-b5a3-517dfd17327b-kube-api-access-hmql8\") pod \"network-check-target-fmkt5\" (UID: \"ce6513b3-2ba0-40fd-b5a3-517dfd17327b\") " pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:12:55.361543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.361465 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcxv2\" (UniqueName: \"kubernetes.io/projected/acb383fe-f128-49fe-b261-c2d7f5d4300f-kube-api-access-rcxv2\") pod \"service-ca-5fffc97bdf-s59nj\" (UID: \"acb383fe-f128-49fe-b261-c2d7f5d4300f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" Nov 02 21:12:55.404259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404230 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404260 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-var-lib-kubelet\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404275 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cnibin\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404291 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-node-log\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404338 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-node-log\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404351 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-cnibin\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404351 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-var-lib-kubelet\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404375 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404405 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404424 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-netns\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404430 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404439 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-etc-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404446 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404463 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-log-socket\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404472 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-etc-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404495 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404503 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-log-socket\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404552 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-netns\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404573 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-netd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404588 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysconfig\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404606 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da7cd6-2547-44ec-a373-e4af9726b87b-crio-socket\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404628 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-socket-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404637 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-netd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404650 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysconfig\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404651 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f56336c-5149-4436-8bdd-28b2d1b1573f-hosts-file\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404697 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85da7cd6-2547-44ec-a373-e4af9726b87b-crio-socket\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404702 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404714 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-socket-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404715 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f56336c-5149-4436-8bdd-28b2d1b1573f-hosts-file\") pod \"node-resolver-pdftq\" (UID: \"0f56336c-5149-4436-8bdd-28b2d1b1573f\") " pod="openshift-dns/node-resolver-pdftq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404728 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-conf\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404756 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404777 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-kubelet\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404796 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-kubelet\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404827 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404840 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-sysctl-conf\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404871 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-kubelet\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404877 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.404885 4510 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404903 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404911 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-modprobe-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404937 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-kubelet\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404939 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-sys\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.404962 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.904946753 +0000 UTC m=+1.881244506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : secret "metrics-daemon-secret" not found Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404978 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-sys\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404992 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-modprobe-d\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.404987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.404995 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-hostroot\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405020 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-device-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405037 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-hostroot\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405039 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-kubelet-config\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405063 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-system-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405071 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-device-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405078 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-multus\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405082 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-kubelet-config\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405102 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-ovn\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405117 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-systemd\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405110 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-system-cni-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405142 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-lib-modules\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405151 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-multus\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405157 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-multus-certs\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405174 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-multus-certs\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405150 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-ovn\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405190 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-system-cni-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405207 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-conf-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405218 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-systemd\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405230 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-host-slash\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405240 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-conf-dir\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405239 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-lib-modules\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405257 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-system-cni-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405272 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8bb07a2-a355-485e-a0f3-b48d5d8f9862-host-slash\") pod \"iptables-alerter-fbktz\" (UID: \"b8bb07a2-a355-485e-a0f3-b48d5d8f9862\") " pod="openshift-network-operator/iptables-alerter-fbktz" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405297 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-registration-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405313 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-sys-fs\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405328 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-cnibin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405341 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-systemd-units\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405342 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-registration-dir\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405357 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-sys-fs\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405360 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405378 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-cnibin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405379 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405387 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f0f955fe-25c7-49e5-8c6b-a9abea532bc1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qhkj4\" (UID: \"f0f955fe-25c7-49e5-8c6b-a9abea532bc1\") " pod="openshift-multus/multus-additional-cni-plugins-qhkj4" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405408 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-socket-dir-parent\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405430 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-systemd-units\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405433 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f84f0d81-e665-4b14-9680-c2dbae153f43-host\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.405436 4510 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405463 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-multus-socket-dir-parent\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405470 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f84f0d81-e665-4b14-9680-c2dbae153f43-host\") pod \"node-ca-f6955\" (UID: \"f84f0d81-e665-4b14-9680-c2dbae153f43\") " pod="openshift-image-registry/node-ca-f6955" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405484 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-netns\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.405505 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.905489073 +0000 UTC m=+1.881786827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405554 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-run-netns\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405569 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405593 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405608 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-kubernetes\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405623 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-run\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405645 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-k8s-cni-cncf-io\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405669 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-host\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.405673 4510 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405694 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-run\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405682 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-etc-kubernetes\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405715 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ac06bad-5657-47d1-9f15-87fd58f9945f-host\") pod \"tuned-q4ljn\" (UID: \"5ac06bad-5657-47d1-9f15-87fd58f9945f\") " pod="openshift-cluster-node-tuning-operator/tuned-q4ljn" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405717 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-run-k8s-cni-cncf-io\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.405721 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:55.905710422 +0000 UTC m=+1.882008175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405763 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-etc-kubernetes\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405783 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-dbus\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405797 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-slash\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405812 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-etc-kubernetes\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405816 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-systemd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405845 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba030b9f-ec13-45a5-82a9-df4315256051-dbus\") pod \"global-pull-secret-syncer-fdpbr\" (UID: \"ba030b9f-ec13-45a5-82a9-df4315256051\") " pod="kube-system/global-pull-secret-syncer-fdpbr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405847 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-var-lib-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405873 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-bin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405883 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-var-lib-openvswitch\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405890 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-bin\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405874 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-run-systemd\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405848 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-slash\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405911 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.405904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405916 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bfd56dd-b587-42d9-83f8-6788d298b743-host-var-lib-cni-bin\") pod \"multus-ccljr\" (UID: \"7bfd56dd-b587-42d9-83f8-6788d298b743\") " pod="openshift-multus/multus-ccljr" Nov 02 21:12:55.407727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405948 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.407727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405955 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66fda8b-8c6a-480a-9f42-656b485bc49a-host-cni-bin\") pod \"ovnkube-node-p7wfq\" (UID: \"c66fda8b-8c6a-480a-9f42-656b485bc49a\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:55.407727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.405982 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.407727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.406035 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd4b2790-68ec-4dd1-8085-1aeb5fe3521e-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j562\" (UID: \"dd4b2790-68ec-4dd1-8085-1aeb5fe3521e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j562" Nov 02 21:12:55.407727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.406066 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85da7cd6-2547-44ec-a373-e4af9726b87b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.507099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.507074 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:12:55.908992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.908959 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:55.909142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.909018 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:55.909142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:55.909048 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:55.909142 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.909108 4510 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:55.909234 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.909160 4510 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:55.909234 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.909166 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:56.90915 +0000 UTC m=+2.885447766 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : secret "metrics-daemon-secret" not found Nov 02 21:12:55.909234 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.909168 4510 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:55.909234 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.909194 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:56.909184234 +0000 UTC m=+2.885481988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:55.909234 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:55.909224 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:56.909212005 +0000 UTC m=+2.885509758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:56.229550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:56.229467 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pdftq_0f56336c-5149-4436-8bdd-28b2d1b1573f/dns-node-resolver/0.log" Nov 02 21:12:56.916412 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:56.916340 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:56.916581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:56.916412 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:56.916581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:56.916442 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:56.916581 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:56.916477 4510 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:56.916581 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:56.916557 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:58.916538906 +0000 UTC m=+4.892836675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : secret "metrics-daemon-secret" not found Nov 02 21:12:56.916581 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:56.916567 4510 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:56.916787 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:56.916567 4510 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 21:12:56.916787 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:56.916616 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:12:58.916601904 +0000 UTC m=+4.892899662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:56.916787 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:56.916648 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls podName:df44d132-0233-4799-9b13-6c5c314fe8c2 nodeName:}" failed. No retries permitted until 2026-11-02 21:12:58.916635086 +0000 UTC m=+4.892932852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls") pod "dns-default-rrj8p" (UID: "df44d132-0233-4799-9b13-6c5c314fe8c2") : secret "dns-default-metrics-tls" not found Nov 02 21:12:57.192985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:57.192906 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-s59nj" event={"ID":"acb383fe-f128-49fe-b261-c2d7f5d4300f","Type":"ContainerStarted","Data":"b666692d02ba22b77d5e44246ab95594bb14a795cb43f0b15a2b9e9aec0a34a0"} Nov 02 21:12:57.430272 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:57.430246 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f6955_f84f0d81-e665-4b14-9680-c2dbae153f43/node-ca/0.log" Nov 02 21:12:58.430458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.430414 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tmmm8_b1c32e61-e26d-4b32-9a55-4a6770edd655/serve-healthcheck-canary/0.log" Nov 02 21:12:58.831678 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.831610 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-tfnj7_ef5d3dd9-d000-43dc-bf45-fbae72cdd767/migrator/0.log" Nov 02 21:12:58.919097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.919074 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:12:58.931876 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.931856 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:58.931990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.931885 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:12:58.931990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.931914 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:12:58.932292 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:58.932270 4510 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 21:12:58.932373 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:58.932339 4510 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:12:58.932475 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:58.932462 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls podName:85da7cd6-2547-44ec-a373-e4af9726b87b nodeName:}" failed. No retries permitted until 2026-11-02 21:13:02.932442292 +0000 UTC m=+8.908740059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4c9pl" (UID: "85da7cd6-2547-44ec-a373-e4af9726b87b") : secret "insights-runtime-extractor-tls" not found Nov 02 21:12:58.932558 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:12:58.932499 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:13:02.932480147 +0000 UTC m=+8.908777916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : secret "metrics-daemon-secret" not found Nov 02 21:12:58.932799 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.932747 4510 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:12:58.935284 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:58.935256 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df44d132-0233-4799-9b13-6c5c314fe8c2-metrics-tls\") pod \"dns-default-rrj8p\" (UID: \"df44d132-0233-4799-9b13-6c5c314fe8c2\") " pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:59.103570 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:59.103505 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rrj8p" Nov 02 21:12:59.467826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:12:59.467796 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rrj8p"] Nov 02 21:12:59.475669 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:12:59.475642 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf44d132_0233_4799_9b13_6c5c314fe8c2.slice/crio-e58788930c6665751dc843ef543d45ff6e07978c510c3be0d6d6ba11838e42e5 WatchSource:0}: Error finding container e58788930c6665751dc843ef543d45ff6e07978c510c3be0d6d6ba11838e42e5: Status 404 returned error can't find the container with id e58788930c6665751dc843ef543d45ff6e07978c510c3be0d6d6ba11838e42e5 Nov 02 21:13:00.204473 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:00.204438 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrj8p" event={"ID":"df44d132-0233-4799-9b13-6c5c314fe8c2","Type":"ContainerStarted","Data":"e58788930c6665751dc843ef543d45ff6e07978c510c3be0d6d6ba11838e42e5"} Nov 02 21:13:01.303442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:01.303407 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:13:02.211883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:02.211843 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrj8p" event={"ID":"df44d132-0233-4799-9b13-6c5c314fe8c2","Type":"ContainerStarted","Data":"4ffec3cadd7428cc13b7ddec85667383d49098396fa576cd66750229a9714f9b"} Nov 02 21:13:02.212258 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:02.211908 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rrj8p" event={"ID":"df44d132-0233-4799-9b13-6c5c314fe8c2","Type":"ContainerStarted","Data":"eb033a899b6647ddca9fadf02a22e54e25beef59bdbd7253cc48bac91d814378"} Nov 02 21:13:02.212258 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:02.211991 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rrj8p" Nov 02 21:13:02.957784 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:02.957748 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:13:02.958271 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:02.957797 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:13:02.958271 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:13:02.957895 4510 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:13:02.958271 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:13:02.957945 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs podName:cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9 nodeName:}" failed. No retries permitted until 2026-11-02 21:13:10.957932737 +0000 UTC m=+16.934230490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs") pod "network-metrics-daemon-b26fp" (UID: "cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9") : secret "metrics-daemon-secret" not found Nov 02 21:13:02.960125 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:02.960102 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85da7cd6-2547-44ec-a373-e4af9726b87b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4c9pl\" (UID: \"85da7cd6-2547-44ec-a373-e4af9726b87b\") " pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:13:03.005335 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:03.005299 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4c9pl" Nov 02 21:13:03.367175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:03.367150 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4c9pl"] Nov 02 21:13:03.408004 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:03.407962 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85da7cd6_2547_44ec_a373_e4af9726b87b.slice/crio-9707f0a27efba08fc3d2051d2e532e888af9a2d7d492abe4505e6b57fdcb843b WatchSource:0}: Error finding container 9707f0a27efba08fc3d2051d2e532e888af9a2d7d492abe4505e6b57fdcb843b: Status 404 returned error can't find the container with id 9707f0a27efba08fc3d2051d2e532e888af9a2d7d492abe4505e6b57fdcb843b Nov 02 21:13:04.220942 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:04.220905 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c9pl" event={"ID":"85da7cd6-2547-44ec-a373-e4af9726b87b","Type":"ContainerStarted","Data":"849bef9f82b918a7fd0ee4a5e3ec676d134160e0dc1a59d37ea88e2524f8014e"} Nov 02 21:13:04.221342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:04.220969 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c9pl" event={"ID":"85da7cd6-2547-44ec-a373-e4af9726b87b","Type":"ContainerStarted","Data":"9707f0a27efba08fc3d2051d2e532e888af9a2d7d492abe4505e6b57fdcb843b"} Nov 02 21:13:04.753436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:04.753416 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:13:04.753649 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:04.753634 4510 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:13:05.225441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:05.225410 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c9pl" event={"ID":"85da7cd6-2547-44ec-a373-e4af9726b87b","Type":"ContainerStarted","Data":"3ea3eb0fc2b90739b8a9219431fbb84dd455c1c4ffdde2b70554711f3c9f410f"} Nov 02 21:13:07.234548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:07.234502 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4c9pl" event={"ID":"85da7cd6-2547-44ec-a373-e4af9726b87b","Type":"ContainerStarted","Data":"9afdfb8df289afd0ea4ccf8dc4d9d1430a02cddeeb5bbbc649c2710615e12a7d"} Nov 02 21:13:11.011249 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:11.011211 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:13:11.013426 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:11.013406 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9-metrics-certs\") pod \"network-metrics-daemon-b26fp\" (UID: \"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9\") " pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:13:11.099566 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:11.099537 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b26fp" Nov 02 21:13:11.461749 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:11.461713 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b26fp"] Nov 02 21:13:11.471142 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:11.471116 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd552cb4_b31b_4df0_aa52_0ebab4f8d2c9.slice/crio-351e85364bd4115f7c378e20d636a2585ede662d8ece971e6d9fb7074713e2b1 WatchSource:0}: Error finding container 351e85364bd4115f7c378e20d636a2585ede662d8ece971e6d9fb7074713e2b1: Status 404 returned error can't find the container with id 351e85364bd4115f7c378e20d636a2585ede662d8ece971e6d9fb7074713e2b1 Nov 02 21:13:12.007077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:12.007053 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7wfq" Nov 02 21:13:12.218158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:12.218129 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rrj8p" Nov 02 21:13:12.249998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:12.249970 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b26fp" event={"ID":"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9","Type":"ContainerStarted","Data":"351e85364bd4115f7c378e20d636a2585ede662d8ece971e6d9fb7074713e2b1"} Nov 02 21:13:13.256168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:13.256136 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b26fp" event={"ID":"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9","Type":"ContainerStarted","Data":"72c48774aa585c40fa12e2e15b04422da8866c61786e83054d44adf84468e2ee"} Nov 02 21:13:13.256455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:13.256207 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b26fp" event={"ID":"cd552cb4-b31b-4df0-aa52-0ebab4f8d2c9","Type":"ContainerStarted","Data":"d55480635917c5d5f6da3b0a25a586a37f597e3d1782b7bed3d6a7db5b6328aa"} Nov 02 21:13:15.528906 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:13:15.528848 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:13:22.074113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:22.074083 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wcgz4" Nov 02 21:13:22.908107 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:22.908072 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw"] Nov 02 21:13:22.912233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:22.912217 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:22.975332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:22.975302 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c516d4c-2849-47a2-b48e-83dd60d924d3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:22.975332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:22.975332 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthxc\" (UniqueName: \"kubernetes.io/projected/2c516d4c-2849-47a2-b48e-83dd60d924d3-kube-api-access-hthxc\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:22.975480 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:22.975356 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c516d4c-2849-47a2-b48e-83dd60d924d3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:22.975480 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:22.975403 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c516d4c-2849-47a2-b48e-83dd60d924d3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.006788 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.006757 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 21:13:23.006899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.006785 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:13:23.006899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.006824 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2tgbf\"" Nov 02 21:13:23.007011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.006976 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 21:13:23.007011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.006986 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:13:23.007011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.006992 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:13:23.007697 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.007680 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw"] Nov 02 21:13:23.053959 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.053939 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf"] Nov 02 21:13:23.057402 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.057374 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b9p7l"] Nov 02 21:13:23.057493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.057478 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.060711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.060695 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.076555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076536 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.076883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076592 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7gx\" (UniqueName: \"kubernetes.io/projected/018583a9-9602-4e9e-9799-9ae2b89707db-kube-api-access-tl7gx\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.076883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076614 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-tls\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.076883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076634 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c516d4c-2849-47a2-b48e-83dd60d924d3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.076883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076750 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10194992-9c92-47ed-92a6-3075cabbf27d-metrics-client-ca\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.076883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076783 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-sys\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.076883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076816 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/018583a9-9602-4e9e-9799-9ae2b89707db-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.076883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076850 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076922 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-root\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076942 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.076978 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c516d4c-2849-47a2-b48e-83dd60d924d3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077030 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8kk7\" (UniqueName: \"kubernetes.io/projected/10194992-9c92-47ed-92a6-3075cabbf27d-kube-api-access-l8kk7\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077094 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c516d4c-2849-47a2-b48e-83dd60d924d3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077123 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthxc\" (UniqueName: \"kubernetes.io/projected/2c516d4c-2849-47a2-b48e-83dd60d924d3-kube-api-access-hthxc\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077130 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c516d4c-2849-47a2-b48e-83dd60d924d3-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077150 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-textfile\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.077209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077176 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-wtmp\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.077550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077214 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/018583a9-9602-4e9e-9799-9ae2b89707db-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.077550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077268 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.077550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.077308 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.079251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.079228 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c516d4c-2849-47a2-b48e-83dd60d924d3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.079365 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.079349 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c516d4c-2849-47a2-b48e-83dd60d924d3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.107295 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.107259 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 21:13:23.107394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.107375 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 21:13:23.107723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.107707 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7g26r\"" Nov 02 21:13:23.107803 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.107711 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 21:13:23.108445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.108426 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf"] Nov 02 21:13:23.108886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.108845 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:13:23.108886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.108861 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:13:23.109020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.108957 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:13:23.109066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.109030 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hbtfx\"" Nov 02 21:13:23.129300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.129278 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthxc\" (UniqueName: \"kubernetes.io/projected/2c516d4c-2849-47a2-b48e-83dd60d924d3-kube-api-access-hthxc\") pod \"openshift-state-metrics-7bd7489957-8shvw\" (UID: \"2c516d4c-2849-47a2-b48e-83dd60d924d3\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.177650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177595 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8kk7\" (UniqueName: \"kubernetes.io/projected/10194992-9c92-47ed-92a6-3075cabbf27d-kube-api-access-l8kk7\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.177650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177633 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-textfile\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.177800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177658 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-wtmp\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.177800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177693 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/018583a9-9602-4e9e-9799-9ae2b89707db-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.177800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177724 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.177800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177748 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177805 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177837 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7gx\" (UniqueName: \"kubernetes.io/projected/018583a9-9602-4e9e-9799-9ae2b89707db-kube-api-access-tl7gx\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177875 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-wtmp\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177942 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-textfile\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.177985 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-tls\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178021 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10194992-9c92-47ed-92a6-3075cabbf27d-metrics-client-ca\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178047 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-sys\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178082 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/018583a9-9602-4e9e-9799-9ae2b89707db-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178106 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178135 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-root\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178158 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178252 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-sys\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178493 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/018583a9-9602-4e9e-9799-9ae2b89707db-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178575 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/018583a9-9602-4e9e-9799-9ae2b89707db-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178575 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.178621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178579 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10194992-9c92-47ed-92a6-3075cabbf27d-root\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.178797 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.178692 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10194992-9c92-47ed-92a6-3075cabbf27d-metrics-client-ca\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.179080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.179043 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.180082 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.180066 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.180311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.180294 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.180924 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.180902 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10194992-9c92-47ed-92a6-3075cabbf27d-node-exporter-tls\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.181109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.181088 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/018583a9-9602-4e9e-9799-9ae2b89707db-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.226866 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.226839 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" Nov 02 21:13:23.231046 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.230754 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7gx\" (UniqueName: \"kubernetes.io/projected/018583a9-9602-4e9e-9799-9ae2b89707db-kube-api-access-tl7gx\") pod \"kube-state-metrics-5dc8d7d89b-2gspf\" (UID: \"018583a9-9602-4e9e-9799-9ae2b89707db\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.231046 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.230924 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8kk7\" (UniqueName: \"kubernetes.io/projected/10194992-9c92-47ed-92a6-3075cabbf27d-kube-api-access-l8kk7\") pod \"node-exporter-b9p7l\" (UID: \"10194992-9c92-47ed-92a6-3075cabbf27d\") " pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.373835 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.373803 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" Nov 02 21:13:23.381226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.381179 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9p7l" Nov 02 21:13:23.390992 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:23.390968 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10194992_9c92_47ed_92a6_3075cabbf27d.slice/crio-ab6b68ce56a6b4761e9d773ee7d08d755e7f90642fa5beec98e7b52b7fe1f0d5 WatchSource:0}: Error finding container ab6b68ce56a6b4761e9d773ee7d08d755e7f90642fa5beec98e7b52b7fe1f0d5: Status 404 returned error can't find the container with id ab6b68ce56a6b4761e9d773ee7d08d755e7f90642fa5beec98e7b52b7fe1f0d5 Nov 02 21:13:23.591402 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.591317 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw"] Nov 02 21:13:23.599896 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:23.599866 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c516d4c_2849_47a2_b48e_83dd60d924d3.slice/crio-ca2db4409e7c56acb6bd4527598f505acbfd73f0bf13505ec03430be853c8da1 WatchSource:0}: Error finding container ca2db4409e7c56acb6bd4527598f505acbfd73f0bf13505ec03430be853c8da1: Status 404 returned error can't find the container with id ca2db4409e7c56acb6bd4527598f505acbfd73f0bf13505ec03430be853c8da1 Nov 02 21:13:23.743374 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:23.743349 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf"] Nov 02 21:13:23.751604 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:23.751581 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018583a9_9602_4e9e_9799_9ae2b89707db.slice/crio-baa49957f7c505934e67a52aefcc1cad3fcfc4f463d12bf28b7094cc582809e3 WatchSource:0}: Error finding container baa49957f7c505934e67a52aefcc1cad3fcfc4f463d12bf28b7094cc582809e3: Status 404 returned error can't find the container with id baa49957f7c505934e67a52aefcc1cad3fcfc4f463d12bf28b7094cc582809e3 Nov 02 21:13:24.289939 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:24.289898 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9p7l" event={"ID":"10194992-9c92-47ed-92a6-3075cabbf27d","Type":"ContainerStarted","Data":"ab6b68ce56a6b4761e9d773ee7d08d755e7f90642fa5beec98e7b52b7fe1f0d5"} Nov 02 21:13:24.291084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:24.291056 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" event={"ID":"018583a9-9602-4e9e-9799-9ae2b89707db","Type":"ContainerStarted","Data":"baa49957f7c505934e67a52aefcc1cad3fcfc4f463d12bf28b7094cc582809e3"} Nov 02 21:13:24.292697 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:24.292676 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" event={"ID":"2c516d4c-2849-47a2-b48e-83dd60d924d3","Type":"ContainerStarted","Data":"80c9143674b4f093b8ba312644aeefcb72ece1c6fe98954fc67fa7ff47e31d68"} Nov 02 21:13:24.292789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:24.292723 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" event={"ID":"2c516d4c-2849-47a2-b48e-83dd60d924d3","Type":"ContainerStarted","Data":"aff37557b6331d87b89c9a6c5b33e749ff0186a6765fac79ceda4bf7ac95e4cd"} Nov 02 21:13:24.292789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:24.292752 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" event={"ID":"2c516d4c-2849-47a2-b48e-83dd60d924d3","Type":"ContainerStarted","Data":"ca2db4409e7c56acb6bd4527598f505acbfd73f0bf13505ec03430be853c8da1"} Nov 02 21:13:25.299557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:25.299475 4510 generic.go:356] "Generic (PLEG): container finished" podID="10194992-9c92-47ed-92a6-3075cabbf27d" containerID="323b1976898913249cf8dbd353c4f8bb9e82803234827b3c2f930ee6924eca63" exitCode=0 Nov 02 21:13:25.299557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:25.299527 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9p7l" event={"ID":"10194992-9c92-47ed-92a6-3075cabbf27d","Type":"ContainerDied","Data":"323b1976898913249cf8dbd353c4f8bb9e82803234827b3c2f930ee6924eca63"} Nov 02 21:13:26.305496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:26.305436 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9p7l" event={"ID":"10194992-9c92-47ed-92a6-3075cabbf27d","Type":"ContainerStarted","Data":"7b0a2584fd4868aa25bfcebd3fe0f644971675fe10eae1c0d7866ddd4ea6d316"} Nov 02 21:13:26.305926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:26.305542 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9p7l" event={"ID":"10194992-9c92-47ed-92a6-3075cabbf27d","Type":"ContainerStarted","Data":"33f7cbe600ebae0148cc726519c8c33e085cf00f494bb8e3b100bfd44b272c77"} Nov 02 21:13:26.307426 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:26.307398 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" event={"ID":"018583a9-9602-4e9e-9799-9ae2b89707db","Type":"ContainerStarted","Data":"d9074bc69f03a6b2e96cbabcd1e05e9b564e2523e393d04379a66c878621cce5"} Nov 02 21:13:26.307542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:26.307454 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" event={"ID":"018583a9-9602-4e9e-9799-9ae2b89707db","Type":"ContainerStarted","Data":"5a8711db851bec402698db6a489ff97faa607c9007c6a58685ba62c33c14cdd7"} Nov 02 21:13:26.310157 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:26.310131 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" event={"ID":"2c516d4c-2849-47a2-b48e-83dd60d924d3","Type":"ContainerStarted","Data":"30be4311131d12ab34de59f0a9e43f85520da40355375ee5c233a2968629fddf"} Nov 02 21:13:26.432801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:26.432737 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b9p7l" podStartSLOduration=2.768369467 podStartE2EDuration="4.432724299s" totalImagesPullingTime="1.586687198s" totalInitContainerRuntime="77.667634ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:26.431969597 +0000 UTC m=+32.408267372" watchObservedRunningTime="2026-11-02 21:13:26.432724299 +0000 UTC m=+32.409022119" Nov 02 21:13:26.542308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:26.542259 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8shvw" podStartSLOduration=2.9158371 podStartE2EDuration="4.542245395s" totalImagesPullingTime="1.626408295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:26.542218345 +0000 UTC m=+32.518516121" watchObservedRunningTime="2026-11-02 21:13:26.542245395 +0000 UTC m=+32.518543164" Nov 02 21:13:27.316275 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.316236 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" event={"ID":"018583a9-9602-4e9e-9799-9ae2b89707db","Type":"ContainerStarted","Data":"2ab8308aa45a83ce8bcc041c6010ec81b8f936646308fba17cdb7b2e4e1bd3e8"} Nov 02 21:13:27.333945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.333922 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:13:27.369321 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.369301 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.389309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.389290 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:13:27.412913 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.412892 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.412927 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.412958 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.412975 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413178 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413099 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413208 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413329 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413396 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413351 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-web-config\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413396 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413373 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-config-out\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413413 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413456 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413510 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpc5\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-kube-api-access-ldpc5\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.413670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.413655 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-config-volume\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.420360 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.420342 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:13:27.420475 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.420371 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:13:27.420545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.420513 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-stkhv\"" Nov 02 21:13:27.436036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.435984 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2gspf" podStartSLOduration=3.07441532 podStartE2EDuration="5.435970192s" totalImagesPullingTime="2.361554872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:27.435014687 +0000 UTC m=+33.411312468" watchObservedRunningTime="2026-11-02 21:13:27.435970192 +0000 UTC m=+33.412267971" Nov 02 21:13:27.485343 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.485313 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:13:27.485442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.485349 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:13:27.485442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.485351 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:13:27.485442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.485387 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:13:27.485442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.485401 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:13:27.485442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.485436 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:13:27.485639 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.485461 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:13:27.514147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514127 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514151 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514167 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514185 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514221 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514242 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-web-config\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514270 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-config-out\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514575 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514554 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514686 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.514807 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514792 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpc5\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-kube-api-access-ldpc5\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.515676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.514741 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.515676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.515645 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.516545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.515927 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-config-volume\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.516545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.515983 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.516545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.516017 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.517226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.517200 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.517651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.517632 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.517767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.517642 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.517844 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.517652 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.517909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.517887 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.518031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.518003 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-web-config\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.518143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.518127 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-config-out\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.518193 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.518170 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.519017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.518997 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-config-volume\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.519017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.519011 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.568792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.568740 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpc5\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-kube-api-access-ldpc5\") pod \"alertmanager-main-0\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.673464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.673428 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8bb997b84-t778t"] Nov 02 21:13:27.703634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.703610 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.717920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.717898 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.717927 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c270975-023a-4326-9de8-9ad0a11dc490-metrics-client-ca\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.717945 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6jv\" (UniqueName: \"kubernetes.io/projected/6c270975-023a-4326-9de8-9ad0a11dc490-kube-api-access-5l6jv\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.717976 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.718127 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c270975-023a-4326-9de8-9ad0a11dc490-tmp\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.718196 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718262 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.718235 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-grpc-tls\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718310 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.718270 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-tls\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.718354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.718327 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.738144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.738109 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8bb997b84-t778t"] Nov 02 21:13:27.752317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.752278 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 21:13:27.752317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.752278 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eof60bqo7mkdj\"" Nov 02 21:13:27.752317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.752303 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-dwrwr\"" Nov 02 21:13:27.752317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.752283 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 21:13:27.752555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.752278 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 21:13:27.752555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.752310 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 21:13:27.752555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.752315 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 21:13:27.769081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.769057 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:13:27.818493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818469 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb"] Nov 02 21:13:27.818738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818720 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c270975-023a-4326-9de8-9ad0a11dc490-tmp\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.818849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818762 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.818849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818796 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-grpc-tls\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.818849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818832 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-tls\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.819021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818863 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.819021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818895 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.819021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818921 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c270975-023a-4326-9de8-9ad0a11dc490-metrics-client-ca\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.819021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.818947 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6jv\" (UniqueName: \"kubernetes.io/projected/6c270975-023a-4326-9de8-9ad0a11dc490-kube-api-access-5l6jv\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.819021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.819005 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.819247 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.819074 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c270975-023a-4326-9de8-9ad0a11dc490-tmp\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.820908 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.820861 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c270975-023a-4326-9de8-9ad0a11dc490-metrics-client-ca\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.821853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.821827 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-grpc-tls\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.821936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.821853 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.822869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.822850 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.823154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.823133 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.823776 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.823750 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.824547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.824507 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6c270975-023a-4326-9de8-9ad0a11dc490-secret-thanos-querier-tls\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.849337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.847075 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:27.876370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.876345 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb"] Nov 02 21:13:27.877545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.877434 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6jv\" (UniqueName: \"kubernetes.io/projected/6c270975-023a-4326-9de8-9ad0a11dc490-kube-api-access-5l6jv\") pod \"thanos-querier-8bb997b84-t778t\" (UID: \"6c270975-023a-4326-9de8-9ad0a11dc490\") " pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:27.882859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.882838 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b989958c5-kgdg8"] Nov 02 21:13:27.895465 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.895436 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 21:13:27.895593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.895564 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w987g\"" Nov 02 21:13:27.908163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.908148 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920076 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-audit-log\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920103 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-client-ca-bundle\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920127 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ce4cdffc-6153-440f-99e7-1ba21b4821cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kmxb\" (UID: \"ce4cdffc-6153-440f-99e7-1ba21b4821cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:27.920309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920182 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-secret-metrics-server-tls\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920345 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920414 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920385 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-metrics-server-audit-profiles\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920459 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920422 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-secret-metrics-server-client-certs\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920481 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjjk\" (UniqueName: \"kubernetes.io/projected/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-kube-api-access-qqjjk\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:27.920567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.920540 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7lfj\" (UniqueName: \"kubernetes.io/projected/ce4cdffc-6153-440f-99e7-1ba21b4821cc-kube-api-access-j7lfj\") pod \"monitoring-plugin-77f4f76594-5kmxb\" (UID: \"ce4cdffc-6153-440f-99e7-1ba21b4821cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:27.940694 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.940672 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b989958c5-kgdg8"] Nov 02 21:13:27.957138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.957110 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 21:13:27.957221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.957170 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 21:13:27.996025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.996004 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 21:13:27.996135 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.996039 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qcrp7\"" Nov 02 21:13:27.996229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.996208 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 21:13:27.996330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:27.996263 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3v6jl92ais13f\"" Nov 02 21:13:28.021252 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021225 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.021344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021256 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-metrics-server-audit-profiles\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.021344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021277 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-secret-metrics-server-client-certs\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.021344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021316 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjjk\" (UniqueName: \"kubernetes.io/projected/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-kube-api-access-qqjjk\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.021344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021342 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7lfj\" (UniqueName: \"kubernetes.io/projected/ce4cdffc-6153-440f-99e7-1ba21b4821cc-kube-api-access-j7lfj\") pod \"monitoring-plugin-77f4f76594-5kmxb\" (UID: \"ce4cdffc-6153-440f-99e7-1ba21b4821cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:28.021555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021360 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-audit-log\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.021555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021373 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-client-ca-bundle\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.021555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021404 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ce4cdffc-6153-440f-99e7-1ba21b4821cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kmxb\" (UID: \"ce4cdffc-6153-440f-99e7-1ba21b4821cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:28.021715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021608 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-secret-metrics-server-tls\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.021886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.021842 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-audit-log\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.022229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.022208 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.022593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.022569 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-metrics-server-audit-profiles\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.023918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.023899 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ce4cdffc-6153-440f-99e7-1ba21b4821cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kmxb\" (UID: \"ce4cdffc-6153-440f-99e7-1ba21b4821cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:28.024100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.024079 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-secret-metrics-server-tls\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.024228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.024215 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-client-ca-bundle\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.024264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.024234 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-secret-metrics-server-client-certs\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.074923 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.074877 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7lfj\" (UniqueName: \"kubernetes.io/projected/ce4cdffc-6153-440f-99e7-1ba21b4821cc-kube-api-access-j7lfj\") pod \"monitoring-plugin-77f4f76594-5kmxb\" (UID: \"ce4cdffc-6153-440f-99e7-1ba21b4821cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:28.075010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.074921 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjjk\" (UniqueName: \"kubernetes.io/projected/8bbdf546-cf6d-4a48-bb1c-ee01e66d2847-kube-api-access-qqjjk\") pod \"metrics-server-7b989958c5-kgdg8\" (UID: \"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847\") " pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.097880 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.097863 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:28.173912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.173889 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:28.217173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.217148 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:13:28.228978 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:28.228946 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35015558_99b2_4cf0_994d_e35a42f10299.slice/crio-471d9d50b817e50dac60222b4af3d48bc222cbb85f6264eb3e5fa93f72041a50 WatchSource:0}: Error finding container 471d9d50b817e50dac60222b4af3d48bc222cbb85f6264eb3e5fa93f72041a50: Status 404 returned error can't find the container with id 471d9d50b817e50dac60222b4af3d48bc222cbb85f6264eb3e5fa93f72041a50 Nov 02 21:13:28.231373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.231351 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:28.325232 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.325162 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerStarted","Data":"471d9d50b817e50dac60222b4af3d48bc222cbb85f6264eb3e5fa93f72041a50"} Nov 02 21:13:28.493742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.493710 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8bb997b84-t778t"] Nov 02 21:13:28.501387 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:28.501360 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c270975_023a_4326_9de8_9ad0a11dc490.slice/crio-02f9eee18c91edef40069793f3d8667f34069884f2b95c2d4f7d4d20f66dca19 WatchSource:0}: Error finding container 02f9eee18c91edef40069793f3d8667f34069884f2b95c2d4f7d4d20f66dca19: Status 404 returned error can't find the container with id 02f9eee18c91edef40069793f3d8667f34069884f2b95c2d4f7d4d20f66dca19 Nov 02 21:13:28.554473 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.554446 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb"] Nov 02 21:13:28.562980 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:28.562943 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4cdffc_6153_440f_99e7_1ba21b4821cc.slice/crio-f62b08d89644583e4af47033947c6014e16a536da878251a1d48d9efbf5d6323 WatchSource:0}: Error finding container f62b08d89644583e4af47033947c6014e16a536da878251a1d48d9efbf5d6323: Status 404 returned error can't find the container with id f62b08d89644583e4af47033947c6014e16a536da878251a1d48d9efbf5d6323 Nov 02 21:13:28.623476 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:28.623446 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b989958c5-kgdg8"] Nov 02 21:13:28.631467 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:28.631444 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbdf546_cf6d_4a48_bb1c_ee01e66d2847.slice/crio-30db6a016d70d5c9e56badf70110028751c8f63a4d6a922036465b593127e18b WatchSource:0}: Error finding container 30db6a016d70d5c9e56badf70110028751c8f63a4d6a922036465b593127e18b: Status 404 returned error can't find the container with id 30db6a016d70d5c9e56badf70110028751c8f63a4d6a922036465b593127e18b Nov 02 21:13:29.331933 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:29.331901 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" event={"ID":"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847","Type":"ContainerStarted","Data":"30db6a016d70d5c9e56badf70110028751c8f63a4d6a922036465b593127e18b"} Nov 02 21:13:29.333105 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:29.333080 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" event={"ID":"ce4cdffc-6153-440f-99e7-1ba21b4821cc","Type":"ContainerStarted","Data":"f62b08d89644583e4af47033947c6014e16a536da878251a1d48d9efbf5d6323"} Nov 02 21:13:29.334296 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:29.334274 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" event={"ID":"6c270975-023a-4326-9de8-9ad0a11dc490","Type":"ContainerStarted","Data":"02f9eee18c91edef40069793f3d8667f34069884f2b95c2d4f7d4d20f66dca19"} Nov 02 21:13:31.306050 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:31.306021 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fmkt5" Nov 02 21:13:31.351282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:31.351244 4510 generic.go:356] "Generic (PLEG): container finished" podID="35015558-99b2-4cf0-994d-e35a42f10299" containerID="a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75" exitCode=0 Nov 02 21:13:31.351429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:31.351314 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75"} Nov 02 21:13:32.357137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.357049 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" event={"ID":"8bbdf546-cf6d-4a48-bb1c-ee01e66d2847","Type":"ContainerStarted","Data":"70b8a5914c65148de1456738f0c150acec21ff7db5ad4d1723bcbe713a280305"} Nov 02 21:13:32.358648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.358624 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" event={"ID":"ce4cdffc-6153-440f-99e7-1ba21b4821cc","Type":"ContainerStarted","Data":"b550e9e42905e8876464b6e90c2da5c420ef7ac7652ffc95d70b55b52c5d0b6a"} Nov 02 21:13:32.358781 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.358743 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:32.360926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.360904 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" event={"ID":"6c270975-023a-4326-9de8-9ad0a11dc490","Type":"ContainerStarted","Data":"8a83a5b01590d293fb1dcdc0fdf86f95d5fc837e28d425c7d871e13d9558d9e1"} Nov 02 21:13:32.361010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.360950 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" event={"ID":"6c270975-023a-4326-9de8-9ad0a11dc490","Type":"ContainerStarted","Data":"7c9d67da6b1e222478c13473be9566315012b499eebdb003d6dfaf6b48be5aaf"} Nov 02 21:13:32.361010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.360981 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" event={"ID":"6c270975-023a-4326-9de8-9ad0a11dc490","Type":"ContainerStarted","Data":"8944eddef2ac39172b8161e58b5832e26e602de422dfd204a1d9bccfd24bb497"} Nov 02 21:13:32.363498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.363474 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" Nov 02 21:13:32.462879 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.462842 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" podStartSLOduration=2.280307403 podStartE2EDuration="5.462830904s" totalImagesPullingTime="3.182523501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:32.462396008 +0000 UTC m=+38.438693782" watchObservedRunningTime="2026-11-02 21:13:32.462830904 +0000 UTC m=+38.439128673" Nov 02 21:13:32.566982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:32.566916 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kmxb" podStartSLOduration=2.315759575 podStartE2EDuration="5.566898722s" totalImagesPullingTime="3.251139147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:32.566624323 +0000 UTC m=+38.542922099" watchObservedRunningTime="2026-11-02 21:13:32.566898722 +0000 UTC m=+38.543196498" Nov 02 21:13:34.092270 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.092188 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:13:34.095931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.095914 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.144764 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.144733 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 21:13:34.144764 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.144752 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 21:13:34.176168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176139 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176168 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176185 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176213 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176285 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176313 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176510 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176356 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176510 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176429 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176510 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176445 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176510 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176491 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176510 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjqk\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-kube-api-access-ckjqk\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176564 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-config\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176579 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176595 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176622 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176700 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176736 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176756 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176774 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.176851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.176815 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.186991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.186972 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 21:13:34.186991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.186977 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 21:13:34.187136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.187050 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 21:13:34.237939 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.237917 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 21:13:34.237939 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.237920 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 21:13:34.238147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.237988 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 21:13:34.238147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.238018 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 21:13:34.238147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.238022 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 21:13:34.238147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.238050 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9b8wn\"" Nov 02 21:13:34.238147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.238060 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 21:13:34.238147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.238101 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 21:13:34.238147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.238107 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 21:13:34.238357 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.238161 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2t3814gpelvlt\"" Nov 02 21:13:34.277804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277773 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.277931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277817 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.277931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277839 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.277931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277869 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.277931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277898 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.277931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277926 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277961 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.277983 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278004 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278044 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278066 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278087 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278110 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278151 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278183 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278217 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278241 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjqk\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-kube-api-access-ckjqk\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278265 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-config\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278288 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.278634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.278318 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.279674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.279640 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.279787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.279719 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.280975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.280955 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.282297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.282250 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.282297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.282267 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.282752 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.282729 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.282850 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.282768 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.282995 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.282949 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.283140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.283113 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.283231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.283143 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.283353 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.283328 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.284735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.284713 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 21:13:34.285184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.285162 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.285314 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.285167 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-config\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.285314 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.285230 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.285472 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.285299 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.285606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.285581 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.285840 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.285822 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.286389 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.286373 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:13:34.286715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.286693 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.289240 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.289217 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.331870 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.331847 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjqk\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-kube-api-access-ckjqk\") pod \"prometheus-k8s-0\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.373011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.372982 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerStarted","Data":"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8"} Nov 02 21:13:34.373127 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.373058 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerStarted","Data":"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642"} Nov 02 21:13:34.373127 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.373098 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerStarted","Data":"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df"} Nov 02 21:13:34.373234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.373135 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerStarted","Data":"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73"} Nov 02 21:13:34.373234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.373171 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerStarted","Data":"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab"} Nov 02 21:13:34.373234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.373214 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerStarted","Data":"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301"} Nov 02 21:13:34.376061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.376036 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" event={"ID":"6c270975-023a-4326-9de8-9ad0a11dc490","Type":"ContainerStarted","Data":"875bc23e51a2c57ecb1e7b59db338f5d874f57a0ddeb5a54036e80d9496033ca"} Nov 02 21:13:34.376170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.376102 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" event={"ID":"6c270975-023a-4326-9de8-9ad0a11dc490","Type":"ContainerStarted","Data":"8b0d66af1adccd8a93d1891ca4381fd5333f877bd40afc3ef8f729ea416de509"} Nov 02 21:13:34.376170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.376141 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" event={"ID":"6c270975-023a-4326-9de8-9ad0a11dc490","Type":"ContainerStarted","Data":"95eb6c923a66655169564d4ac8230d5790072a7d70ec9cace50f0c7cc5142e4e"} Nov 02 21:13:34.376263 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.376233 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:34.421281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.421264 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:34.486856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.486812 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.908428454 podStartE2EDuration="7.486794991s" totalImagesPullingTime="4.557846923s" totalInitContainerRuntime="20.519614ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:34.486030944 +0000 UTC m=+40.462328718" watchObservedRunningTime="2026-11-02 21:13:34.486794991 +0000 UTC m=+40.463092766" Nov 02 21:13:34.596265 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.596212 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" podStartSLOduration=2.788584576 podStartE2EDuration="7.596198897s" totalImagesPullingTime="4.807614321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:34.595556097 +0000 UTC m=+40.571853873" watchObservedRunningTime="2026-11-02 21:13:34.596198897 +0000 UTC m=+40.572496711" Nov 02 21:13:34.846650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:34.846573 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:13:34.854507 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:13:34.854482 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f77b81f_869d_4429_b4c7_bf72d974c32d.slice/crio-72322758d2ac7c14b5c8f6707e1635da23073250e9dc78f639d91fefdb7b6eb1 WatchSource:0}: Error finding container 72322758d2ac7c14b5c8f6707e1635da23073250e9dc78f639d91fefdb7b6eb1: Status 404 returned error can't find the container with id 72322758d2ac7c14b5c8f6707e1635da23073250e9dc78f639d91fefdb7b6eb1 Nov 02 21:13:35.381057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:35.381021 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerID="2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083" exitCode=0 Nov 02 21:13:35.381467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:35.381068 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083"} Nov 02 21:13:35.381467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:35.381170 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerStarted","Data":"72322758d2ac7c14b5c8f6707e1635da23073250e9dc78f639d91fefdb7b6eb1"} Nov 02 21:13:39.399965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.399880 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerStarted","Data":"e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507"} Nov 02 21:13:39.400428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.399983 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerStarted","Data":"930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01"} Nov 02 21:13:39.400428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.400029 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerStarted","Data":"2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606"} Nov 02 21:13:39.400428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.400068 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerStarted","Data":"4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd"} Nov 02 21:13:39.400428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.400117 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerStarted","Data":"bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690"} Nov 02 21:13:39.400428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.400154 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerStarted","Data":"e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4"} Nov 02 21:13:39.421544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.421509 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:13:39.652292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:39.652194 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.1546696929999998 podStartE2EDuration="5.652181737s" totalImagesPullingTime="3.483674534s" totalInitContainerRuntime="13.83751ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:13:39.605892167 +0000 UTC m=+45.582189969" watchObservedRunningTime="2026-11-02 21:13:39.652181737 +0000 UTC m=+45.628479511" Nov 02 21:13:40.388257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:40.388228 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8bb997b84-t778t" Nov 02 21:13:45.533986 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:13:45.533942 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:13:48.231766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:48.231736 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:13:48.232334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:13:48.231795 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:14:08.236874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:08.236843 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:14:08.240476 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:08.240453 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b989958c5-kgdg8" Nov 02 21:14:15.251397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:15.251365 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:15.269464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:15.269440 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:15.534538 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:15.534423 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:14:15.550152 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:15.550123 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:17.543889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:17.543808 4510 generic.go:356] "Generic (PLEG): container finished" podID="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" containerID="886cec361c7d267242dcac4e416442c57157cee3f96cb277ab02a9e8bb38ad96" exitCode=0 Nov 02 21:14:17.543889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:17.543865 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" event={"ID":"dcec323e-b13a-4bc2-b83d-246ada8bbbb3","Type":"ContainerDied","Data":"886cec361c7d267242dcac4e416442c57157cee3f96cb277ab02a9e8bb38ad96"} Nov 02 21:14:18.680015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.679990 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:14:18.764147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.764121 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\") pod \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " Nov 02 21:14:18.764250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.764182 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\" (UniqueName: \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\") pod \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " Nov 02 21:14:18.764250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.764209 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\") pod \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " Nov 02 21:14:18.764328 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.764264 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\") pod \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\" (UID: \"dcec323e-b13a-4bc2-b83d-246ada8bbbb3\") " Nov 02 21:14:18.764574 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.764545 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle" pod "dcec323e-b13a-4bc2-b83d-246ada8bbbb3" (UID: "dcec323e-b13a-4bc2-b83d-246ada8bbbb3"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:18.764655 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.764610 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle" pod "dcec323e-b13a-4bc2-b83d-246ada8bbbb3" (UID: "dcec323e-b13a-4bc2-b83d-246ada8bbbb3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:18.764655 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.764624 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path" pod "dcec323e-b13a-4bc2-b83d-246ada8bbbb3" (UID: "dcec323e-b13a-4bc2-b83d-246ada8bbbb3"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:18.766399 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.766374 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv" pod "dcec323e-b13a-4bc2-b83d-246ada8bbbb3" (UID: "dcec323e-b13a-4bc2-b83d-246ada8bbbb3"). InnerVolumeSpecName "kube-api-access-kqkbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:18.865536 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.865493 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqkbv\" (UniqueName: \"kubernetes.io/projected/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-kube-api-access-kqkbv\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:18.865682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.865548 4510 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-trusted-ca-bundle\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:18.865682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.865566 4510 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-archives-path\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:18.865682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:18.865576 4510 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcec323e-b13a-4bc2-b83d-246ada8bbbb3-service-ca-bundle\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:19.558993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:19.558954 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" event={"ID":"dcec323e-b13a-4bc2-b83d-246ada8bbbb3","Type":"ContainerDied","Data":"b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657"} Nov 02 21:14:19.559243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:19.559006 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b24d932dcb795520f433014a5c6ca01462c77239ab1162800a460b3d3e896657" Nov 02 21:14:19.559243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:19.558970 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-szkkx-zrfdx" Nov 02 21:14:25.701475 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:25.701445 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tmmm8_b1c32e61-e26d-4b32-9a55-4a6770edd655/serve-healthcheck-canary/0.log" Nov 02 21:14:30.602718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:30.602687 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:14:30.603184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:30.603132 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="alertmanager" containerID="cri-o://1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301" gracePeriod=120 Nov 02 21:14:30.603282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:30.603166 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="prom-label-proxy" containerID="cri-o://102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8" gracePeriod=120 Nov 02 21:14:30.603341 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:30.603320 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-metric" containerID="cri-o://b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642" gracePeriod=120 Nov 02 21:14:30.603428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:30.603397 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-web" containerID="cri-o://aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73" gracePeriod=120 Nov 02 21:14:30.603428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:30.603417 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="config-reloader" containerID="cri-o://283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab" gracePeriod=120 Nov 02 21:14:30.604754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:30.603718 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy" containerID="cri-o://f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df" gracePeriod=120 Nov 02 21:14:31.604607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604574 4510 generic.go:356] "Generic (PLEG): container finished" podID="35015558-99b2-4cf0-994d-e35a42f10299" containerID="102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8" exitCode=0 Nov 02 21:14:31.604607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604601 4510 generic.go:356] "Generic (PLEG): container finished" podID="35015558-99b2-4cf0-994d-e35a42f10299" containerID="f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df" exitCode=0 Nov 02 21:14:31.604607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604609 4510 generic.go:356] "Generic (PLEG): container finished" podID="35015558-99b2-4cf0-994d-e35a42f10299" containerID="283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab" exitCode=0 Nov 02 21:14:31.604607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604616 4510 generic.go:356] "Generic (PLEG): container finished" podID="35015558-99b2-4cf0-994d-e35a42f10299" containerID="1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301" exitCode=0 Nov 02 21:14:31.605088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604646 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8"} Nov 02 21:14:31.605088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604710 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df"} Nov 02 21:14:31.605088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604736 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab"} Nov 02 21:14:31.605088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.604760 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301"} Nov 02 21:14:31.864470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.864446 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:31.973370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973337 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-web\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973380 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973409 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-cluster-tls-config\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973429 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-metrics-client-ca\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973583 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-web-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-web-config\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973646 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-config-out\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-config-out\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973678 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-kube-api-access-ldpc5\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-kube-api-access-ldpc5\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973745 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-main-tls\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973773 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-tls-assets\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-tls-assets\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.973878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973781 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-metrics-client-ca" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:31.973878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973829 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-main-db\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.974174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973877 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.974174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973918 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-trusted-ca-bundle\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.974174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.973946 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-config-volume\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-config-volume\") pod \"35015558-99b2-4cf0-994d-e35a42f10299\" (UID: \"35015558-99b2-4cf0-994d-e35a42f10299\") " Nov 02 21:14:31.974337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.974202 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-metrics-client-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:31.974337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.974206 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-main-db" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:31.975017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.974989 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-trusted-ca-bundle" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:31.976572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.976545 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-web" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:31.977225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.977113 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-metric" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:31.977225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.977200 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-main-tls" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:31.977391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.977269 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-kube-api-access-ldpc5" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "kube-api-access-ldpc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:31.977686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.977643 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-config-out" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:31.977897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.977862 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-config-volume" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:31.977981 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.977927 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-tls-assets" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:31.978153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.978126 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:31.981917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.981898 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-cluster-tls-config" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:31.988381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:31.988359 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-web-config" pod "35015558-99b2-4cf0-994d-e35a42f10299" (UID: "35015558-99b2-4cf0-994d-e35a42f10299"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:32.075216 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075185 4510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075216 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075209 4510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075216 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075222 4510 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-cluster-tls-config\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075233 4510 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-web-config\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075242 4510 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-config-out\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075251 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldpc5\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-kube-api-access-ldpc5\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075259 4510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-main-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075269 4510 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35015558-99b2-4cf0-994d-e35a42f10299-tls-assets\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075276 4510 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-main-db\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075285 4510 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075293 4510 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35015558-99b2-4cf0-994d-e35a42f10299-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.075410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.075303 4510 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35015558-99b2-4cf0-994d-e35a42f10299-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:32.611228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.611151 4510 generic.go:356] "Generic (PLEG): container finished" podID="35015558-99b2-4cf0-994d-e35a42f10299" containerID="b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642" exitCode=0 Nov 02 21:14:32.611228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.611176 4510 generic.go:356] "Generic (PLEG): container finished" podID="35015558-99b2-4cf0-994d-e35a42f10299" containerID="aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73" exitCode=0 Nov 02 21:14:32.611228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.611224 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.611746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.611218 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642"} Nov 02 21:14:32.611746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.611359 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73"} Nov 02 21:14:32.611746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.611387 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35015558-99b2-4cf0-994d-e35a42f10299","Type":"ContainerDied","Data":"471d9d50b817e50dac60222b4af3d48bc222cbb85f6264eb3e5fa93f72041a50"} Nov 02 21:14:32.611746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.611430 4510 scope.go:122] "RemoveContainer" containerID="102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8" Nov 02 21:14:32.621092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.621073 4510 scope.go:122] "RemoveContainer" containerID="b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642" Nov 02 21:14:32.628417 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.628391 4510 scope.go:122] "RemoveContainer" containerID="f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df" Nov 02 21:14:32.635567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.635551 4510 scope.go:122] "RemoveContainer" containerID="aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73" Nov 02 21:14:32.642779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.642759 4510 scope.go:122] "RemoveContainer" containerID="283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab" Nov 02 21:14:32.649649 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.649629 4510 scope.go:122] "RemoveContainer" containerID="1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301" Nov 02 21:14:32.657813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.657799 4510 scope.go:122] "RemoveContainer" containerID="a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75" Nov 02 21:14:32.666635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.666606 4510 scope.go:122] "RemoveContainer" containerID="102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8" Nov 02 21:14:32.666888 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:32.666868 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8\": container with ID starting with 102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8 not found: ID does not exist" containerID="102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8" Nov 02 21:14:32.666953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.666897 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8"} err="failed to get container status \"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8\": rpc error: code = NotFound desc = could not find container \"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8\": container with ID starting with 102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8 not found: ID does not exist" Nov 02 21:14:32.666953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.666928 4510 scope.go:122] "RemoveContainer" containerID="b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642" Nov 02 21:14:32.667158 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:32.667143 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642\": container with ID starting with b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642 not found: ID does not exist" containerID="b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642" Nov 02 21:14:32.667213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667162 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642"} err="failed to get container status \"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642\": rpc error: code = NotFound desc = could not find container \"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642\": container with ID starting with b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642 not found: ID does not exist" Nov 02 21:14:32.667213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667174 4510 scope.go:122] "RemoveContainer" containerID="f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df" Nov 02 21:14:32.667397 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:32.667381 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df\": container with ID starting with f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df not found: ID does not exist" containerID="f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df" Nov 02 21:14:32.667441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667401 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df"} err="failed to get container status \"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df\": rpc error: code = NotFound desc = could not find container \"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df\": container with ID starting with f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df not found: ID does not exist" Nov 02 21:14:32.667441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667415 4510 scope.go:122] "RemoveContainer" containerID="aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73" Nov 02 21:14:32.667650 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:32.667634 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73\": container with ID starting with aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73 not found: ID does not exist" containerID="aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73" Nov 02 21:14:32.667774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667655 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73"} err="failed to get container status \"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73\": rpc error: code = NotFound desc = could not find container \"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73\": container with ID starting with aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73 not found: ID does not exist" Nov 02 21:14:32.667774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667667 4510 scope.go:122] "RemoveContainer" containerID="283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab" Nov 02 21:14:32.667907 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:32.667892 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab\": container with ID starting with 283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab not found: ID does not exist" containerID="283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab" Nov 02 21:14:32.667947 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667909 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab"} err="failed to get container status \"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab\": rpc error: code = NotFound desc = could not find container \"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab\": container with ID starting with 283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab not found: ID does not exist" Nov 02 21:14:32.667947 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.667921 4510 scope.go:122] "RemoveContainer" containerID="1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301" Nov 02 21:14:32.668126 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:32.668111 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301\": container with ID starting with 1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301 not found: ID does not exist" containerID="1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301" Nov 02 21:14:32.668164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668128 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301"} err="failed to get container status \"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301\": rpc error: code = NotFound desc = could not find container \"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301\": container with ID starting with 1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301 not found: ID does not exist" Nov 02 21:14:32.668164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668138 4510 scope.go:122] "RemoveContainer" containerID="a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75" Nov 02 21:14:32.668341 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:32.668328 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75\": container with ID starting with a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75 not found: ID does not exist" containerID="a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75" Nov 02 21:14:32.668383 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668342 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75"} err="failed to get container status \"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75\": rpc error: code = NotFound desc = could not find container \"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75\": container with ID starting with a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75 not found: ID does not exist" Nov 02 21:14:32.668383 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668353 4510 scope.go:122] "RemoveContainer" containerID="102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8" Nov 02 21:14:32.668590 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668559 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8"} err="failed to get container status \"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8\": rpc error: code = NotFound desc = could not find container \"102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8\": container with ID starting with 102f55c36e606333095348874c3de1f82991ddadd60e044946962c54f038b2b8 not found: ID does not exist" Nov 02 21:14:32.668648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668591 4510 scope.go:122] "RemoveContainer" containerID="b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642" Nov 02 21:14:32.668801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668787 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642"} err="failed to get container status \"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642\": rpc error: code = NotFound desc = could not find container \"b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642\": container with ID starting with b5b948123f764dd95bf9e883ffc58d482df0a5768cb09bbe0792e11acb17d642 not found: ID does not exist" Nov 02 21:14:32.668801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668801 4510 scope.go:122] "RemoveContainer" containerID="f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df" Nov 02 21:14:32.668992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668976 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df"} err="failed to get container status \"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df\": rpc error: code = NotFound desc = could not find container \"f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df\": container with ID starting with f7de4186f5a7b61221f62fa037b3e2b23ee5dda1e9b6e8e7e550d1e9dad450df not found: ID does not exist" Nov 02 21:14:32.668992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.668991 4510 scope.go:122] "RemoveContainer" containerID="aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73" Nov 02 21:14:32.669195 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.669175 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73"} err="failed to get container status \"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73\": rpc error: code = NotFound desc = could not find container \"aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73\": container with ID starting with aaaf5c30754ca75170548e1631b58469f8ad30dc94e58a2226a62f916fc93c73 not found: ID does not exist" Nov 02 21:14:32.669243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.669195 4510 scope.go:122] "RemoveContainer" containerID="283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab" Nov 02 21:14:32.669413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.669395 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab"} err="failed to get container status \"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab\": rpc error: code = NotFound desc = could not find container \"283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab\": container with ID starting with 283b3fc2a767f773ee7f6e9ada9a721d527d0d62a3960e53d3ab7801d9b390ab not found: ID does not exist" Nov 02 21:14:32.669413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.669412 4510 scope.go:122] "RemoveContainer" containerID="1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301" Nov 02 21:14:32.669686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.669669 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301"} err="failed to get container status \"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301\": rpc error: code = NotFound desc = could not find container \"1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301\": container with ID starting with 1392e10932833c017f75c0b8e3ab6e5d0c8ae682ea3ab2dbe6618fe32c42a301 not found: ID does not exist" Nov 02 21:14:32.669686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.669685 4510 scope.go:122] "RemoveContainer" containerID="a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75" Nov 02 21:14:32.669902 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.669887 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75"} err="failed to get container status \"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75\": rpc error: code = NotFound desc = could not find container \"a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75\": container with ID starting with a1f455f674deb38192eb2c6315e2958f30b573acf22c6db4cf81288b27e74c75 not found: ID does not exist" Nov 02 21:14:32.811913 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.811886 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.860921 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.860989 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861587 4510 cpu_manager.go:414] "containerMap: removing container" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="prom-label-proxy" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861604 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="prom-label-proxy" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861625 4510 cpu_manager.go:414] "containerMap: removing container" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861632 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861656 4510 cpu_manager.go:414] "containerMap: removing container" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="config-reloader" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861666 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="config-reloader" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861725 4510 cpu_manager.go:414] "containerMap: removing container" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-web" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861734 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-web" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861744 4510 cpu_manager.go:414] "containerMap: removing container" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-metric" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861752 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-metric" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861769 4510 cpu_manager.go:414] "containerMap: removing container" podUID="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" containerName="insights-gathering" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861776 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" containerName="insights-gathering" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861793 4510 cpu_manager.go:414] "containerMap: removing container" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="alertmanager" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861800 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="alertmanager" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861814 4510 cpu_manager.go:414] "containerMap: removing container" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="init-config-reloader" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861821 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="init-config-reloader" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861952 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="config-reloader" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861970 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-web" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861985 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy-metric" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.861994 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcec323e-b13a-4bc2-b83d-246ada8bbbb3" containerName="insights-gathering" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.862009 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="alertmanager" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.862019 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="prom-label-proxy" Nov 02 21:14:32.862930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.862028 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="35015558-99b2-4cf0-994d-e35a42f10299" containerName="kube-rbac-proxy" Nov 02 21:14:32.866868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.866839 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.916129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916100 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:14:32.916129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916097 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:14:32.916363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916173 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:14:32.916363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916203 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:14:32.916363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916208 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-stkhv\"" Nov 02 21:14:32.916363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916246 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:14:32.916363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916282 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:14:32.916363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916285 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:14:32.916630 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.916515 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:14:32.962159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.962128 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:14:32.982899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.982870 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-config-volume\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.982899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.982899 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed250005-c680-4d6d-9e28-b99581d03f16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.982923 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-web-config\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.982940 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983013 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983082 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed250005-c680-4d6d-9e28-b99581d03f16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983137 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983204 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983265 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983247 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed250005-c680-4d6d-9e28-b99581d03f16-config-out\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983278 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfw9\" (UniqueName: \"kubernetes.io/projected/ed250005-c680-4d6d-9e28-b99581d03f16-kube-api-access-djfw9\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983300 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983409 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983338 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed250005-c680-4d6d-9e28-b99581d03f16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:32.983459 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:32.983451 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed250005-c680-4d6d-9e28-b99581d03f16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.083698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.083657 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:14:33.083966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.083942 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed250005-c680-4d6d-9e28-b99581d03f16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.083998 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed250005-c680-4d6d-9e28-b99581d03f16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084028 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-config-volume\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084443 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084419 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ed250005-c680-4d6d-9e28-b99581d03f16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084474 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed250005-c680-4d6d-9e28-b99581d03f16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084510 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-web-config\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084557 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084587 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084619 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed250005-c680-4d6d-9e28-b99581d03f16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084688 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084716 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084752 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed250005-c680-4d6d-9e28-b99581d03f16-config-out\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.084809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084787 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfw9\" (UniqueName: \"kubernetes.io/projected/ed250005-c680-4d6d-9e28-b99581d03f16-kube-api-access-djfw9\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.085047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084813 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.085047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.084893 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed250005-c680-4d6d-9e28-b99581d03f16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.085157 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.085069 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed250005-c680-4d6d-9e28-b99581d03f16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.088086 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.088045 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed250005-c680-4d6d-9e28-b99581d03f16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.088616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.088482 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-config-volume\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.089295 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.089247 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.089391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.089292 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed250005-c680-4d6d-9e28-b99581d03f16-config-out\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.089572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.089547 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.089912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.089888 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.090616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.090571 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-web-config\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.090733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.090715 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.091828 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.091809 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ed250005-c680-4d6d-9e28-b99581d03f16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.138413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.138350 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfw9\" (UniqueName: \"kubernetes.io/projected/ed250005-c680-4d6d-9e28-b99581d03f16-kube-api-access-djfw9\") pod \"alertmanager-main-0\" (UID: \"ed250005-c680-4d6d-9e28-b99581d03f16\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.191688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.191663 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:14:33.616214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:33.616189 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:14:33.624181 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:14:33.624158 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded250005_c680_4d6d_9e28_b99581d03f16.slice/crio-481bd8309975720d8d42b31eeffe127bf484fba22fad97515c162737b4819770 WatchSource:0}: Error finding container 481bd8309975720d8d42b31eeffe127bf484fba22fad97515c162737b4819770: Status 404 returned error can't find the container with id 481bd8309975720d8d42b31eeffe127bf484fba22fad97515c162737b4819770 Nov 02 21:14:34.150884 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:34.150852 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35015558-99b2-4cf0-994d-e35a42f10299" path="/var/lib/kubelet/pods/35015558-99b2-4cf0-994d-e35a42f10299/volumes" Nov 02 21:14:34.622095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:34.622067 4510 generic.go:356] "Generic (PLEG): container finished" podID="ed250005-c680-4d6d-9e28-b99581d03f16" containerID="299a5d00626e7781b5964e77cc74ee722633a1cc07363b4c5f6e133c72a8e783" exitCode=0 Nov 02 21:14:34.622493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:34.622106 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerDied","Data":"299a5d00626e7781b5964e77cc74ee722633a1cc07363b4c5f6e133c72a8e783"} Nov 02 21:14:34.622493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:34.622156 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerStarted","Data":"481bd8309975720d8d42b31eeffe127bf484fba22fad97515c162737b4819770"} Nov 02 21:14:35.628779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:35.628747 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerStarted","Data":"9909321d8513e1b4eeb01c8826b74ca643a1e75296eabd5039a7dcdb2493737e"} Nov 02 21:14:35.629144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:35.628806 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerStarted","Data":"298b3e3345f6dc1dcf269224346bff1f23f94879d32b7bd18e75313dd0210f4a"} Nov 02 21:14:35.629144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:35.628834 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerStarted","Data":"debe1f300636d1538627e3af4abad8d2b38a54846d0abc48c9110a1de64480c4"} Nov 02 21:14:35.629144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:35.628861 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerStarted","Data":"e7a075fcb0b51eff0d63671237697e0810eb62682f7b0d6617923216ba03a631"} Nov 02 21:14:35.629144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:35.628885 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerStarted","Data":"ab883717842a5c0d2b6184126461a0ad5ff7068e5ede10a45fa413b56667bca0"} Nov 02 21:14:35.629144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:35.628906 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ed250005-c680-4d6d-9e28-b99581d03f16","Type":"ContainerStarted","Data":"1b95f9bf87af92b24beaa58b9d8ef7eaa99faf763e1a3dcd8d4ad807cb7a245e"} Nov 02 21:14:35.742071 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:35.742027 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.725045446 podStartE2EDuration="3.742014586s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.96914ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:35.740499492 +0000 UTC m=+101.716797267" watchObservedRunningTime="2026-11-02 21:14:35.742014586 +0000 UTC m=+101.718312361" Nov 02 21:14:36.118102 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.118071 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:14:36.118373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.118345 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="prometheus" containerID="cri-o://e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4" gracePeriod=600 Nov 02 21:14:36.118443 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.118394 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="thanos-sidecar" containerID="cri-o://4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd" gracePeriod=600 Nov 02 21:14:36.118493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.118426 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-web" containerID="cri-o://2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606" gracePeriod=600 Nov 02 21:14:36.118493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.118393 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507" gracePeriod=600 Nov 02 21:14:36.118493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.118445 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="config-reloader" containerID="cri-o://bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690" gracePeriod=600 Nov 02 21:14:36.118626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.118376 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy" containerID="cri-o://930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01" gracePeriod=600 Nov 02 21:14:36.635990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.635959 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerID="e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507" exitCode=0 Nov 02 21:14:36.635990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.635984 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerID="930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01" exitCode=0 Nov 02 21:14:36.635990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.635992 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerID="4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd" exitCode=0 Nov 02 21:14:36.635990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.635997 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerID="bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690" exitCode=0 Nov 02 21:14:36.636454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.636003 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerID="e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4" exitCode=0 Nov 02 21:14:36.636454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.636035 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507"} Nov 02 21:14:36.636454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.636104 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01"} Nov 02 21:14:36.636454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.636131 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd"} Nov 02 21:14:36.636454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.636154 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690"} Nov 02 21:14:36.636454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:36.636178 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4"} Nov 02 21:14:37.364934 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.364912 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.422502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422452 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422481 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422641 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422509 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-db\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422641 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422556 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-trusted-ca-bundle\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422641 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422602 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-web-config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-web-config\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422641 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422634 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-kubelet-serving-ca-bundle\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422676 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-1\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422703 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-config-out\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422748 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-tls-assets\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-tls-assets\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422778 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-0\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422806 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-kube-rbac-proxy\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.422842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422836 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-metrics-client-certs\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422863 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-2\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422892 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-kube-api-access-ckjqk\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-kube-api-access-ckjqk\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422927 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-tls\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422960 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-thanos-prometheus-http-client-file\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.422998 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-grpc-tls\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.423053 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-serving-certs-ca-bundle\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.423065 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-trusted-ca-bundle" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.423107 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-metrics-client-ca\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.423143 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-config\") pod \"7f77b81f-869d-4429-b4c7-bf72d974c32d\" (UID: \"7f77b81f-869d-4429-b4c7-bf72d974c32d\") " Nov 02 21:14:37.423663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.423384 4510 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.424228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.423795 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-db" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:37.424228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.423951 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-kubelet-serving-ca-bundle" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:37.424382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.424266 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-0" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:37.424932 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.424907 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-serving-certs-ca-bundle" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:37.425134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.425107 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-thanos-sidecar-tls" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.425734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.425708 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.425819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.425731 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-1" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:37.425889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.425848 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-2" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:37.426000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.425975 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-metrics-client-ca" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:14:37.426286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.426261 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-kube-api-access-ckjqk" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "kube-api-access-ckjqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:37.426408 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.426387 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-config" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.427423 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.427395 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-config-out" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:14:37.428225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.427970 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-grpc-tls" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.428225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.428057 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-metrics-client-certs" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.428225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.428101 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-tls-assets" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:14:37.428225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.428140 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-tls" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.428454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.428416 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-thanos-prometheus-http-client-file" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.428561 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.428540 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-kube-rbac-proxy" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.436173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.436152 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-web-config" pod "7f77b81f-869d-4429-b4c7-bf72d974c32d" (UID: "7f77b81f-869d-4429-b4c7-bf72d974c32d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:14:37.524313 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524292 4510 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524313 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524311 4510 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-metrics-client-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524320 4510 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-config\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524332 4510 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524342 4510 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524351 4510 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-db\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524361 4510 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-web-config\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524370 4510 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524379 4510 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524388 4510 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f77b81f-869d-4429-b4c7-bf72d974c32d-config-out\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524396 4510 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-tls-assets\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524404 4510 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524412 4510 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-kube-rbac-proxy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524420 4510 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-metrics-client-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524427 4510 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f77b81f-869d-4429-b4c7-bf72d974c32d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524435 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckjqk\" (UniqueName: \"kubernetes.io/projected/7f77b81f-869d-4429-b4c7-bf72d974c32d-kube-api-access-ckjqk\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524817 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524444 4510 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524817 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524452 4510 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.524817 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.524460 4510 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f77b81f-869d-4429-b4c7-bf72d974c32d-secret-grpc-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:14:37.644900 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.644871 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerID="2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606" exitCode=0 Nov 02 21:14:37.645291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.644950 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606"} Nov 02 21:14:37.645291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.645001 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.645291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.645021 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f77b81f-869d-4429-b4c7-bf72d974c32d","Type":"ContainerDied","Data":"72322758d2ac7c14b5c8f6707e1635da23073250e9dc78f639d91fefdb7b6eb1"} Nov 02 21:14:37.645291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.645054 4510 scope.go:122] "RemoveContainer" containerID="e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507" Nov 02 21:14:37.653372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.653346 4510 scope.go:122] "RemoveContainer" containerID="930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01" Nov 02 21:14:37.662167 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.662148 4510 scope.go:122] "RemoveContainer" containerID="2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606" Nov 02 21:14:37.670154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.670136 4510 scope.go:122] "RemoveContainer" containerID="4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd" Nov 02 21:14:37.682512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.682495 4510 scope.go:122] "RemoveContainer" containerID="bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690" Nov 02 21:14:37.695405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.695373 4510 scope.go:122] "RemoveContainer" containerID="e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4" Nov 02 21:14:37.702218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.702199 4510 scope.go:122] "RemoveContainer" containerID="2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083" Nov 02 21:14:37.708391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.708372 4510 scope.go:122] "RemoveContainer" containerID="e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507" Nov 02 21:14:37.708721 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:37.708701 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507\": container with ID starting with e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507 not found: ID does not exist" containerID="e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507" Nov 02 21:14:37.708780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.708731 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507"} err="failed to get container status \"e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507\": rpc error: code = NotFound desc = could not find container \"e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507\": container with ID starting with e02e9bff75247fa3fca42f93c5648bf63cd68dbccfdc79eec372db9b6ceb0507 not found: ID does not exist" Nov 02 21:14:37.708780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.708748 4510 scope.go:122] "RemoveContainer" containerID="930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01" Nov 02 21:14:37.709021 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:37.709005 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01\": container with ID starting with 930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01 not found: ID does not exist" containerID="930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01" Nov 02 21:14:37.709064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709027 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01"} err="failed to get container status \"930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01\": rpc error: code = NotFound desc = could not find container \"930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01\": container with ID starting with 930a34d094bff46b2d1fd92026201adbf439009882592ac431f7361e573cea01 not found: ID does not exist" Nov 02 21:14:37.709064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709041 4510 scope.go:122] "RemoveContainer" containerID="2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606" Nov 02 21:14:37.709264 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:37.709246 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606\": container with ID starting with 2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606 not found: ID does not exist" containerID="2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606" Nov 02 21:14:37.709306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709270 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606"} err="failed to get container status \"2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606\": rpc error: code = NotFound desc = could not find container \"2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606\": container with ID starting with 2670588952fc9edb306d1106d7688ed943868700a4ff82e14dab6aefee9bc606 not found: ID does not exist" Nov 02 21:14:37.709306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709285 4510 scope.go:122] "RemoveContainer" containerID="4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd" Nov 02 21:14:37.709512 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:37.709496 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd\": container with ID starting with 4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd not found: ID does not exist" containerID="4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd" Nov 02 21:14:37.709572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709515 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd"} err="failed to get container status \"4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd\": rpc error: code = NotFound desc = could not find container \"4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd\": container with ID starting with 4830e877b6ba44a1fb857777a9c1f99b23daa0001034f63aa7f24f6a1dc991dd not found: ID does not exist" Nov 02 21:14:37.709572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709543 4510 scope.go:122] "RemoveContainer" containerID="bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690" Nov 02 21:14:37.709752 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:37.709737 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690\": container with ID starting with bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690 not found: ID does not exist" containerID="bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690" Nov 02 21:14:37.709787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709755 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690"} err="failed to get container status \"bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690\": rpc error: code = NotFound desc = could not find container \"bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690\": container with ID starting with bfe266d76ec96592f428f58b8d9da41aae7bd65ed4f928fa6f4b4759df2f3690 not found: ID does not exist" Nov 02 21:14:37.709787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.709768 4510 scope.go:122] "RemoveContainer" containerID="e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4" Nov 02 21:14:37.709999 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:37.709983 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4\": container with ID starting with e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4 not found: ID does not exist" containerID="e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4" Nov 02 21:14:37.710051 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.710000 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4"} err="failed to get container status \"e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4\": rpc error: code = NotFound desc = could not find container \"e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4\": container with ID starting with e10aa428bd0faa1cf8c79a16420656e2ef588565e174c8c0b1c759db6894bee4 not found: ID does not exist" Nov 02 21:14:37.710051 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.710012 4510 scope.go:122] "RemoveContainer" containerID="2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083" Nov 02 21:14:37.710202 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:37.710186 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083\": container with ID starting with 2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083 not found: ID does not exist" containerID="2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083" Nov 02 21:14:37.710236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.710206 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083"} err="failed to get container status \"2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083\": rpc error: code = NotFound desc = could not find container \"2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083\": container with ID starting with 2f528f17d266b16f95074c6c663399a25d78cc13c683e18cae04054bab81d083 not found: ID does not exist" Nov 02 21:14:37.798806 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.798780 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:14:37.802178 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.802157 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:14:37.844651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844625 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:14:37.844949 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844936 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="prometheus" Nov 02 21:14:37.844986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844951 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="prometheus" Nov 02 21:14:37.844986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844971 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="init-config-reloader" Nov 02 21:14:37.844986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844977 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="init-config-reloader" Nov 02 21:14:37.844986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844985 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-thanos" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844992 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-thanos" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.844999 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="thanos-sidecar" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845004 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="thanos-sidecar" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845013 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-web" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845017 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-web" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845023 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845028 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845043 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="config-reloader" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845050 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="config-reloader" Nov 02 21:14:37.845096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845095 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="config-reloader" Nov 02 21:14:37.845358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845105 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="prometheus" Nov 02 21:14:37.845358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845114 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-thanos" Nov 02 21:14:37.845358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845120 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="thanos-sidecar" Nov 02 21:14:37.845358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845126 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy-web" Nov 02 21:14:37.845358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.845131 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" containerName="kube-rbac-proxy" Nov 02 21:14:37.850357 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.850339 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.899615 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.899588 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 21:14:37.899736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.899696 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 21:14:37.899859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.899776 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 21:14:37.899991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.899926 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 21:14:37.900214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900114 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 21:14:37.900214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900128 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 21:14:37.900214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900146 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 21:14:37.900214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900114 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 21:14:37.900214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900173 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 21:14:37.900978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900290 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 21:14:37.900978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900336 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2t3814gpelvlt\"" Nov 02 21:14:37.900978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900408 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 21:14:37.900978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900443 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 21:14:37.900978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.900512 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9b8wn\"" Nov 02 21:14:37.928312 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928287 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928321 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928341 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928357 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928403 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928454 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dkln\" (UniqueName: \"kubernetes.io/projected/0fe423ff-907a-403c-b883-5d87efd5750c-kube-api-access-8dkln\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928484 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928549 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928574 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928597 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928613 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe423ff-907a-403c-b883-5d87efd5750c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928658 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928705 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-config\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928750 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928764 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928789 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe423ff-907a-403c-b883-5d87efd5750c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928807 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.928855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928833 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.929013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928867 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.929013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.928904 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:37.947120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.947055 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 21:14:37.947193 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.947177 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 21:14:37.948404 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:37.948389 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:14:38.030009 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.029985 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-config\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030022 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030038 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030059 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe423ff-907a-403c-b883-5d87efd5750c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030084 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030197 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030235 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030271 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030306 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030347 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030371 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030394 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030424 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030449 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dkln\" (UniqueName: \"kubernetes.io/projected/0fe423ff-907a-403c-b883-5d87efd5750c-kube-api-access-8dkln\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030465 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030478 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030505 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030564 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030602 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030627 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe423ff-907a-403c-b883-5d87efd5750c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.030717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030662 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.031089 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.030761 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.031089 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.031051 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.031200 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.031125 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.031539 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.031499 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.033826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.033263 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.033826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.033511 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0fe423ff-907a-403c-b883-5d87efd5750c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.033826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.033739 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.034025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.033883 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.034650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.034102 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.034650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.034612 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.035049 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.035018 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-config\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.035234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.035208 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.036191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.036138 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.036191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.036148 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.036332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.036290 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.036483 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.036459 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0fe423ff-907a-403c-b883-5d87efd5750c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.036757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.036742 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0fe423ff-907a-403c-b883-5d87efd5750c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.037241 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.037220 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0fe423ff-907a-403c-b883-5d87efd5750c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.086361 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.086340 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dkln\" (UniqueName: \"kubernetes.io/projected/0fe423ff-907a-403c-b883-5d87efd5750c-kube-api-access-8dkln\") pod \"prometheus-k8s-0\" (UID: \"0fe423ff-907a-403c-b883-5d87efd5750c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.150793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.150769 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f77b81f-869d-4429-b4c7-bf72d974c32d" path="/var/lib/kubelet/pods/7f77b81f-869d-4429-b4c7-bf72d974c32d/volumes" Nov 02 21:14:38.175750 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.175728 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:38.603780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.603696 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:14:38.612138 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:14:38.612111 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe423ff_907a_403c_b883_5d87efd5750c.slice/crio-b6c9818dc92bb1b8b53cb387d114e835a1d163f75e74209402e81b9c41d61236 WatchSource:0}: Error finding container b6c9818dc92bb1b8b53cb387d114e835a1d163f75e74209402e81b9c41d61236: Status 404 returned error can't find the container with id b6c9818dc92bb1b8b53cb387d114e835a1d163f75e74209402e81b9c41d61236 Nov 02 21:14:38.650674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:38.650646 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerStarted","Data":"b6c9818dc92bb1b8b53cb387d114e835a1d163f75e74209402e81b9c41d61236"} Nov 02 21:14:39.657800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:39.657765 4510 generic.go:356] "Generic (PLEG): container finished" podID="0fe423ff-907a-403c-b883-5d87efd5750c" containerID="311dc0c1356c67745ea9680725c912248bf4680a3b19234886dafef123916e22" exitCode=0 Nov 02 21:14:39.658211 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:39.657814 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerDied","Data":"311dc0c1356c67745ea9680725c912248bf4680a3b19234886dafef123916e22"} Nov 02 21:14:40.664953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:40.664915 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerStarted","Data":"5aff282a3e8108abb8bf07c786335c29bf430373285c0791768f50089542e4c0"} Nov 02 21:14:40.665407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:40.664991 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerStarted","Data":"a283f2e2b9bdcb941ec9cf050027718da853c80098ae5f61e3958ab6c9ca74a1"} Nov 02 21:14:40.665407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:40.665033 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerStarted","Data":"678e4af61b3f5dca4ca098bc8c1c7e38be22bb4a4f4c36bd977e5c55f3e4bd95"} Nov 02 21:14:40.665407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:40.665073 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerStarted","Data":"01d13addce2121b79b7be426445116dd6297e91f6541162d0a6c4bc075978b6e"} Nov 02 21:14:40.665407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:40.665117 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerStarted","Data":"6615eae282aa58a125edab9f4c438db2cc0d81b4a7652acbf06dbccdff392fa2"} Nov 02 21:14:40.665407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:40.665166 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0fe423ff-907a-403c-b883-5d87efd5750c","Type":"ContainerStarted","Data":"7bdad58a3becc5205fe3d39ca7fd436752e45c216f0697e3bdaa7332a537deda"} Nov 02 21:14:40.781901 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:40.781848 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.761647688 podStartE2EDuration="3.781834785s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.187097ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:14:40.780012991 +0000 UTC m=+106.756310787" watchObservedRunningTime="2026-11-02 21:14:40.781834785 +0000 UTC m=+106.758132560" Nov 02 21:14:43.176031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:14:43.175994 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:14:45.544648 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:14:45.544601 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:15:15.529310 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:15:15.529231 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:15:38.176930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:15:38.176897 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:15:38.192590 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:15:38.192564 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:15:38.893389 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:15:38.893365 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:15:45.530946 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:15:45.530902 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:16:15.528563 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:16:15.528507 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:16:45.532240 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:16:45.532158 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:17:15.528226 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:17:15.528189 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:17:45.533186 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:17:45.533149 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:18:15.529548 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:18:15.529446 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:18:25.229488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.229456 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl"] Nov 02 21:18:25.232799 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.232780 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.327436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.327405 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 21:18:25.327436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.327422 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:18:25.327669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.327482 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7xsc6\"" Nov 02 21:18:25.328729 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.328708 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl"] Nov 02 21:18:25.330478 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.330461 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e121a979-8074-441e-b7d2-240fb701f1a9-tmp\") pod \"jobset-operator-674cd59dc5-2n5hl\" (UID: \"e121a979-8074-441e-b7d2-240fb701f1a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.330564 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.330550 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9dk\" (UniqueName: \"kubernetes.io/projected/e121a979-8074-441e-b7d2-240fb701f1a9-kube-api-access-cs9dk\") pod \"jobset-operator-674cd59dc5-2n5hl\" (UID: \"e121a979-8074-441e-b7d2-240fb701f1a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.431651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.431617 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9dk\" (UniqueName: \"kubernetes.io/projected/e121a979-8074-441e-b7d2-240fb701f1a9-kube-api-access-cs9dk\") pod \"jobset-operator-674cd59dc5-2n5hl\" (UID: \"e121a979-8074-441e-b7d2-240fb701f1a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.431812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.431661 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e121a979-8074-441e-b7d2-240fb701f1a9-tmp\") pod \"jobset-operator-674cd59dc5-2n5hl\" (UID: \"e121a979-8074-441e-b7d2-240fb701f1a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.432017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.432002 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e121a979-8074-441e-b7d2-240fb701f1a9-tmp\") pod \"jobset-operator-674cd59dc5-2n5hl\" (UID: \"e121a979-8074-441e-b7d2-240fb701f1a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.487174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.487114 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9dk\" (UniqueName: \"kubernetes.io/projected/e121a979-8074-441e-b7d2-240fb701f1a9-kube-api-access-cs9dk\") pod \"jobset-operator-674cd59dc5-2n5hl\" (UID: \"e121a979-8074-441e-b7d2-240fb701f1a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.553020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.552993 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" Nov 02 21:18:25.920555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.920481 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl"] Nov 02 21:18:25.929041 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:18:25.929012 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode121a979_8074_441e_b7d2_240fb701f1a9.slice/crio-6f9a5dec4c49f23f94b39e3d1a21f39a7758923955f6c17bed6f01590650152b WatchSource:0}: Error finding container 6f9a5dec4c49f23f94b39e3d1a21f39a7758923955f6c17bed6f01590650152b: Status 404 returned error can't find the container with id 6f9a5dec4c49f23f94b39e3d1a21f39a7758923955f6c17bed6f01590650152b Nov 02 21:18:25.930508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:25.930491 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:18:26.496095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:26.496064 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" event={"ID":"e121a979-8074-441e-b7d2-240fb701f1a9","Type":"ContainerStarted","Data":"6f9a5dec4c49f23f94b39e3d1a21f39a7758923955f6c17bed6f01590650152b"} Nov 02 21:18:30.516090 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:30.516015 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" event={"ID":"e121a979-8074-441e-b7d2-240fb701f1a9","Type":"ContainerStarted","Data":"2dd1ee032cd64449f47a50fe7293fd79ca0441dc5b8cb125a1695c10350711ac"} Nov 02 21:18:30.620799 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:30.620751 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-2n5hl" podStartSLOduration=1.346475826 podStartE2EDuration="5.620738815s" totalImagesPullingTime="4.274262989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:30.620679062 +0000 UTC m=+336.596976859" watchObservedRunningTime="2026-11-02 21:18:30.620738815 +0000 UTC m=+336.597036589" Nov 02 21:18:32.975825 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.975795 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n"] Nov 02 21:18:32.979575 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.979555 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:32.994483 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.994458 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03c11143-d425-4968-81c3-c2d138adcb6e-manifests\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:32.994593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.994491 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:32.994593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.994508 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03c11143-d425-4968-81c3-c2d138adcb6e-tmp\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:32.994663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.994589 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:32.994663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.994630 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhfv\" (UniqueName: \"kubernetes.io/projected/03c11143-d425-4968-81c3-c2d138adcb6e-kube-api-access-gxhfv\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:32.994730 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:32.994664 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-metrics-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.077026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.076977 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 21:18:33.077026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.076986 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 21:18:33.077026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.076992 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 21:18:33.077026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.077026 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 21:18:33.077278 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.077046 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 21:18:33.077278 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.077079 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-mt8kr\"" Nov 02 21:18:33.078393 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.078374 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n"] Nov 02 21:18:33.095601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.095578 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03c11143-d425-4968-81c3-c2d138adcb6e-manifests\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.095700 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.095626 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.095700 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.095646 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03c11143-d425-4968-81c3-c2d138adcb6e-tmp\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.095700 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.095665 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.095700 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.095683 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhfv\" (UniqueName: \"kubernetes.io/projected/03c11143-d425-4968-81c3-c2d138adcb6e-kube-api-access-gxhfv\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.095906 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.095714 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-metrics-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.095906 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.095845 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03c11143-d425-4968-81c3-c2d138adcb6e-manifests\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.098087 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.098057 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03c11143-d425-4968-81c3-c2d138adcb6e-tmp\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.098213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.098192 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-webhook-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.098269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.098217 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-metrics-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.098306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.098267 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/03c11143-d425-4968-81c3-c2d138adcb6e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.154683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.154656 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhfv\" (UniqueName: \"kubernetes.io/projected/03c11143-d425-4968-81c3-c2d138adcb6e-kube-api-access-gxhfv\") pod \"opendatahub-operator-controller-manager-676bcb69b5-hk44n\" (UID: \"03c11143-d425-4968-81c3-c2d138adcb6e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.294728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.294650 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:33.794467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:33.794436 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n"] Nov 02 21:18:33.802014 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:18:33.801982 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c11143_d425_4968_81c3_c2d138adcb6e.slice/crio-23b5caa6555c7ce9e20209d975e385ee4374cc5af35c0ff8b0438274b6b4a501 WatchSource:0}: Error finding container 23b5caa6555c7ce9e20209d975e385ee4374cc5af35c0ff8b0438274b6b4a501: Status 404 returned error can't find the container with id 23b5caa6555c7ce9e20209d975e385ee4374cc5af35c0ff8b0438274b6b4a501 Nov 02 21:18:34.533357 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:34.533318 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" event={"ID":"03c11143-d425-4968-81c3-c2d138adcb6e","Type":"ContainerStarted","Data":"23b5caa6555c7ce9e20209d975e385ee4374cc5af35c0ff8b0438274b6b4a501"} Nov 02 21:18:37.547366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:37.547283 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" event={"ID":"03c11143-d425-4968-81c3-c2d138adcb6e","Type":"ContainerStarted","Data":"63820aece9bc4d475467d490a3b90c2b297beb88204a869cb44e51e5902722fe"} Nov 02 21:18:38.552031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:38.551997 4510 generic.go:356] "Generic (PLEG): container finished" podID="03c11143-d425-4968-81c3-c2d138adcb6e" containerID="63820aece9bc4d475467d490a3b90c2b297beb88204a869cb44e51e5902722fe" exitCode=0 Nov 02 21:18:38.552462 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:38.552087 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" event={"ID":"03c11143-d425-4968-81c3-c2d138adcb6e","Type":"ContainerDied","Data":"63820aece9bc4d475467d490a3b90c2b297beb88204a869cb44e51e5902722fe"} Nov 02 21:18:39.558416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:39.558381 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" event={"ID":"03c11143-d425-4968-81c3-c2d138adcb6e","Type":"ContainerStarted","Data":"c1cb78a5d7c4c275df4401f3f4c82daea6aff8406aa678f6cfdd28e9ae78ba9f"} Nov 02 21:18:39.558807 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:39.558532 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:39.769880 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:39.769782 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" podStartSLOduration=3.036691451 podStartE2EDuration="7.769768093s" totalImagesPullingTime="4.27003534s" totalInitContainerRuntime="463.041302ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:39.723828516 +0000 UTC m=+345.700126317" watchObservedRunningTime="2026-11-02 21:18:39.769768093 +0000 UTC m=+345.746065868" Nov 02 21:18:45.275778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.275743 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4"] Nov 02 21:18:45.281991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.281966 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.342211 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.342182 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4"] Nov 02 21:18:45.378192 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.378164 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 21:18:45.378192 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.378187 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 21:18:45.378501 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.378482 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-42ttj\"" Nov 02 21:18:45.400131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.400086 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcv56\" (UniqueName: \"kubernetes.io/projected/77756a42-43cc-4d6f-a2a7-8c889058c214-kube-api-access-jcv56\") pod \"cert-manager-cainjector-5c57bbd4fc-jvrm4\" (UID: \"77756a42-43cc-4d6f-a2a7-8c889058c214\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.400290 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.400185 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77756a42-43cc-4d6f-a2a7-8c889058c214-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jvrm4\" (UID: \"77756a42-43cc-4d6f-a2a7-8c889058c214\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.500914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.500881 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcv56\" (UniqueName: \"kubernetes.io/projected/77756a42-43cc-4d6f-a2a7-8c889058c214-kube-api-access-jcv56\") pod \"cert-manager-cainjector-5c57bbd4fc-jvrm4\" (UID: \"77756a42-43cc-4d6f-a2a7-8c889058c214\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.501116 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.500939 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77756a42-43cc-4d6f-a2a7-8c889058c214-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jvrm4\" (UID: \"77756a42-43cc-4d6f-a2a7-8c889058c214\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.529775 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:18:45.529695 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:18:45.559650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.559623 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77756a42-43cc-4d6f-a2a7-8c889058c214-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-jvrm4\" (UID: \"77756a42-43cc-4d6f-a2a7-8c889058c214\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.559806 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.559787 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcv56\" (UniqueName: \"kubernetes.io/projected/77756a42-43cc-4d6f-a2a7-8c889058c214-kube-api-access-jcv56\") pod \"cert-manager-cainjector-5c57bbd4fc-jvrm4\" (UID: \"77756a42-43cc-4d6f-a2a7-8c889058c214\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.593368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.593343 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" Nov 02 21:18:45.958315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:45.958282 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4"] Nov 02 21:18:45.967324 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:18:45.967296 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77756a42_43cc_4d6f_a2a7_8c889058c214.slice/crio-29dc063e1bdee52cfbbbebb13b170d06394aa27a201671b3a2e4c08a566cd48d WatchSource:0}: Error finding container 29dc063e1bdee52cfbbbebb13b170d06394aa27a201671b3a2e4c08a566cd48d: Status 404 returned error can't find the container with id 29dc063e1bdee52cfbbbebb13b170d06394aa27a201671b3a2e4c08a566cd48d Nov 02 21:18:46.588236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:46.588197 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" event={"ID":"77756a42-43cc-4d6f-a2a7-8c889058c214","Type":"ContainerStarted","Data":"29dc063e1bdee52cfbbbebb13b170d06394aa27a201671b3a2e4c08a566cd48d"} Nov 02 21:18:50.565098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:50.565065 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-676bcb69b5-hk44n" Nov 02 21:18:50.621505 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:50.621468 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" event={"ID":"77756a42-43cc-4d6f-a2a7-8c889058c214","Type":"ContainerStarted","Data":"3068f6422d9e0c4b4abaf2ab5be16fe20e69b4c99cc20988dc41e73b8785f853"} Nov 02 21:18:50.936466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:50.936364 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" podStartSLOduration=1.567842795 podStartE2EDuration="5.93634462s" totalImagesPullingTime="4.368501825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:50.935004994 +0000 UTC m=+356.911302768" watchObservedRunningTime="2026-11-02 21:18:50.93634462 +0000 UTC m=+356.912642395" Nov 02 21:18:53.140835 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.140802 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jhkf4"] Nov 02 21:18:53.231223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.231193 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jhkf4"] Nov 02 21:18:53.231393 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.231331 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.328172 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.328149 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-skc84\"" Nov 02 21:18:53.374069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.374046 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95d2\" (UniqueName: \"kubernetes.io/projected/5d7da0b5-9d2e-4362-85e8-37980e899b12-kube-api-access-v95d2\") pod \"cert-manager-597b95cb84-jhkf4\" (UID: \"5d7da0b5-9d2e-4362-85e8-37980e899b12\") " pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.374211 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.374120 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d7da0b5-9d2e-4362-85e8-37980e899b12-bound-sa-token\") pod \"cert-manager-597b95cb84-jhkf4\" (UID: \"5d7da0b5-9d2e-4362-85e8-37980e899b12\") " pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.474637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.474545 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95d2\" (UniqueName: \"kubernetes.io/projected/5d7da0b5-9d2e-4362-85e8-37980e899b12-kube-api-access-v95d2\") pod \"cert-manager-597b95cb84-jhkf4\" (UID: \"5d7da0b5-9d2e-4362-85e8-37980e899b12\") " pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.474798 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.474637 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d7da0b5-9d2e-4362-85e8-37980e899b12-bound-sa-token\") pod \"cert-manager-597b95cb84-jhkf4\" (UID: \"5d7da0b5-9d2e-4362-85e8-37980e899b12\") " pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.527539 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.527480 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d7da0b5-9d2e-4362-85e8-37980e899b12-bound-sa-token\") pod \"cert-manager-597b95cb84-jhkf4\" (UID: \"5d7da0b5-9d2e-4362-85e8-37980e899b12\") " pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.527679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.527633 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95d2\" (UniqueName: \"kubernetes.io/projected/5d7da0b5-9d2e-4362-85e8-37980e899b12-kube-api-access-v95d2\") pod \"cert-manager-597b95cb84-jhkf4\" (UID: \"5d7da0b5-9d2e-4362-85e8-37980e899b12\") " pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.560453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.560429 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jhkf4" Nov 02 21:18:53.941902 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:53.941866 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jhkf4"] Nov 02 21:18:53.950090 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:18:53.950067 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d7da0b5_9d2e_4362_85e8_37980e899b12.slice/crio-f7522b1e580185d646dce388ae3b7c836e267e9fa2e10fe346cfedf814f785df WatchSource:0}: Error finding container f7522b1e580185d646dce388ae3b7c836e267e9fa2e10fe346cfedf814f785df: Status 404 returned error can't find the container with id f7522b1e580185d646dce388ae3b7c836e267e9fa2e10fe346cfedf814f785df Nov 02 21:18:54.644746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:54.644708 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jhkf4" event={"ID":"5d7da0b5-9d2e-4362-85e8-37980e899b12","Type":"ContainerStarted","Data":"811315f4c9fe1292090da94bdfab624dee6dea3c0452d8772bf998ee648ab869"} Nov 02 21:18:54.645119 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:54.644758 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jhkf4" event={"ID":"5d7da0b5-9d2e-4362-85e8-37980e899b12","Type":"ContainerStarted","Data":"f7522b1e580185d646dce388ae3b7c836e267e9fa2e10fe346cfedf814f785df"} Nov 02 21:18:54.752850 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:18:54.752798 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jhkf4" podStartSLOduration=1.7527828840000002 podStartE2EDuration="1.752782884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:18:54.751543357 +0000 UTC m=+360.727841122" watchObservedRunningTime="2026-11-02 21:18:54.752782884 +0000 UTC m=+360.729080662" Nov 02 21:19:02.358035 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.358001 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v"] Nov 02 21:19:02.362515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.362494 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.410689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.410657 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 21:19:02.410832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.410688 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 21:19:02.419957 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.419935 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v"] Nov 02 21:19:02.448372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.448315 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b51e51-eb64-40a8-beb8-043c9e9d5b51-tmp\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.449863 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.449831 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b51e51-eb64-40a8-beb8-043c9e9d5b51-tls-certs\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.449995 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.449895 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8msx\" (UniqueName: \"kubernetes.io/projected/76b51e51-eb64-40a8-beb8-043c9e9d5b51-kube-api-access-f8msx\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.499981 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.497956 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 21:19:02.499981 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.498136 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f9p44\"" Nov 02 21:19:02.499981 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.498215 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 21:19:02.550470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.550437 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b51e51-eb64-40a8-beb8-043c9e9d5b51-tls-certs\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.550693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.550479 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8msx\" (UniqueName: \"kubernetes.io/projected/76b51e51-eb64-40a8-beb8-043c9e9d5b51-kube-api-access-f8msx\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.550693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.550542 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b51e51-eb64-40a8-beb8-043c9e9d5b51-tmp\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.552831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.552800 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76b51e51-eb64-40a8-beb8-043c9e9d5b51-tmp\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.553042 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.553019 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b51e51-eb64-40a8-beb8-043c9e9d5b51-tls-certs\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.603940 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.603909 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8msx\" (UniqueName: \"kubernetes.io/projected/76b51e51-eb64-40a8-beb8-043c9e9d5b51-kube-api-access-f8msx\") pod \"kube-auth-proxy-795b66c7fb-fjd9v\" (UID: \"76b51e51-eb64-40a8-beb8-043c9e9d5b51\") " pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:02.674633 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:02.674593 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" Nov 02 21:19:03.044627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:03.044547 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v"] Nov 02 21:19:03.052594 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:19:03.052535 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b51e51_eb64_40a8_beb8_043c9e9d5b51.slice/crio-6785b12510477420ddb1a997a95c1a5893e92eec447da86ea25a7396f3ff2292 WatchSource:0}: Error finding container 6785b12510477420ddb1a997a95c1a5893e92eec447da86ea25a7396f3ff2292: Status 404 returned error can't find the container with id 6785b12510477420ddb1a997a95c1a5893e92eec447da86ea25a7396f3ff2292 Nov 02 21:19:03.684264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:03.684229 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" event={"ID":"76b51e51-eb64-40a8-beb8-043c9e9d5b51","Type":"ContainerStarted","Data":"6785b12510477420ddb1a997a95c1a5893e92eec447da86ea25a7396f3ff2292"} Nov 02 21:19:06.434602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.434568 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll8ss"] Nov 02 21:19:06.438056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.438037 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.486783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.486744 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 21:19:06.533072 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.533043 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 21:19:06.533072 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.533044 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bdwp9\"" Nov 02 21:19:06.533262 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.533081 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 21:19:06.534498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.534479 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll8ss"] Nov 02 21:19:06.587903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.587866 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gv4\" (UniqueName: \"kubernetes.io/projected/b166645b-9667-4588-91bc-40bad8d01fd1-kube-api-access-g5gv4\") pod \"limitador-limitador-74dfcfb894-ll8ss\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.588065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.587916 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b166645b-9667-4588-91bc-40bad8d01fd1-config-file\") pod \"limitador-limitador-74dfcfb894-ll8ss\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.664575 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.664541 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll8ss"] Nov 02 21:19:06.688566 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.688478 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gv4\" (UniqueName: \"kubernetes.io/projected/b166645b-9667-4588-91bc-40bad8d01fd1-kube-api-access-g5gv4\") pod \"limitador-limitador-74dfcfb894-ll8ss\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.688691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.688621 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b166645b-9667-4588-91bc-40bad8d01fd1-config-file\") pod \"limitador-limitador-74dfcfb894-ll8ss\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.689250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.689225 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b166645b-9667-4588-91bc-40bad8d01fd1-config-file\") pod \"limitador-limitador-74dfcfb894-ll8ss\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.699291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.699267 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" event={"ID":"76b51e51-eb64-40a8-beb8-043c9e9d5b51","Type":"ContainerStarted","Data":"52f540726f5956b36bd593af4cf229936ef24340165e485830cfeaab0174bd9b"} Nov 02 21:19:06.740793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.740764 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gv4\" (UniqueName: \"kubernetes.io/projected/b166645b-9667-4588-91bc-40bad8d01fd1-kube-api-access-g5gv4\") pod \"limitador-limitador-74dfcfb894-ll8ss\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.750321 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.750303 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:06.804849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:06.804793 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-795b66c7fb-fjd9v" podStartSLOduration=1.945475082 podStartE2EDuration="4.804776254s" totalImagesPullingTime="2.859301172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:06.80363343 +0000 UTC m=+372.779931206" watchObservedRunningTime="2026-11-02 21:19:06.804776254 +0000 UTC m=+372.781074030" Nov 02 21:19:07.116249 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:07.116215 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll8ss"] Nov 02 21:19:07.124736 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:19:07.124704 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb166645b_9667_4588_91bc_40bad8d01fd1.slice/crio-1f46739b3c8c974c9300ddbd4a6dba3e14d98e4ae9929b28889b373d7d3045f3 WatchSource:0}: Error finding container 1f46739b3c8c974c9300ddbd4a6dba3e14d98e4ae9929b28889b373d7d3045f3: Status 404 returned error can't find the container with id 1f46739b3c8c974c9300ddbd4a6dba3e14d98e4ae9929b28889b373d7d3045f3 Nov 02 21:19:07.704708 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:07.704674 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" event={"ID":"b166645b-9667-4588-91bc-40bad8d01fd1","Type":"ContainerStarted","Data":"1f46739b3c8c974c9300ddbd4a6dba3e14d98e4ae9929b28889b373d7d3045f3"} Nov 02 21:19:09.484965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.484929 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xnb8h"] Nov 02 21:19:09.491419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.491384 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:19:09.491594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.491515 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" Nov 02 21:19:09.496535 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.496502 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.540106 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.540073 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xnb8h"] Nov 02 21:19:09.584778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.584746 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9s9fw\"" Nov 02 21:19:09.586625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.586599 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:19:09.615264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.615241 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj7dt\" (UniqueName: \"kubernetes.io/projected/79af2810-e0dc-4385-8714-2fd71edead77-kube-api-access-pj7dt\") pod \"limitador-limitador-7c9b87d574-zw95m\" (UID: \"79af2810-e0dc-4385-8714-2fd71edead77\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.615401 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.615296 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnvg\" (UniqueName: \"kubernetes.io/projected/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67-kube-api-access-bbnvg\") pod \"authorino-56c6c9ddd5-xnb8h\" (UID: \"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67\") " pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" Nov 02 21:19:09.615466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.615416 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79af2810-e0dc-4385-8714-2fd71edead77-config-file\") pod \"limitador-limitador-7c9b87d574-zw95m\" (UID: \"79af2810-e0dc-4385-8714-2fd71edead77\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.716208 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.716172 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79af2810-e0dc-4385-8714-2fd71edead77-config-file\") pod \"limitador-limitador-7c9b87d574-zw95m\" (UID: \"79af2810-e0dc-4385-8714-2fd71edead77\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.716380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.716224 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj7dt\" (UniqueName: \"kubernetes.io/projected/79af2810-e0dc-4385-8714-2fd71edead77-kube-api-access-pj7dt\") pod \"limitador-limitador-7c9b87d574-zw95m\" (UID: \"79af2810-e0dc-4385-8714-2fd71edead77\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.716380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.716298 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnvg\" (UniqueName: \"kubernetes.io/projected/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67-kube-api-access-bbnvg\") pod \"authorino-56c6c9ddd5-xnb8h\" (UID: \"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67\") " pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" Nov 02 21:19:09.716872 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.716850 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79af2810-e0dc-4385-8714-2fd71edead77-config-file\") pod \"limitador-limitador-7c9b87d574-zw95m\" (UID: \"79af2810-e0dc-4385-8714-2fd71edead77\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.769881 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.769818 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj7dt\" (UniqueName: \"kubernetes.io/projected/79af2810-e0dc-4385-8714-2fd71edead77-kube-api-access-pj7dt\") pod \"limitador-limitador-7c9b87d574-zw95m\" (UID: \"79af2810-e0dc-4385-8714-2fd71edead77\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.779255 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.779229 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnvg\" (UniqueName: \"kubernetes.io/projected/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67-kube-api-access-bbnvg\") pod \"authorino-56c6c9ddd5-xnb8h\" (UID: \"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67\") " pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" Nov 02 21:19:09.807012 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.806985 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" Nov 02 21:19:09.813137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.813108 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:09.822849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:09.822824 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:19:10.191367 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:10.191336 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xnb8h"] Nov 02 21:19:10.200077 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:19:10.200049 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod474b5b1a_9103_4ef9_bc9c_81e41b5cfa67.slice/crio-ea94499f1dca816257770cf3645fa0cf44057106af4d82de6fffaef2d556bb9b WatchSource:0}: Error finding container ea94499f1dca816257770cf3645fa0cf44057106af4d82de6fffaef2d556bb9b: Status 404 returned error can't find the container with id ea94499f1dca816257770cf3645fa0cf44057106af4d82de6fffaef2d556bb9b Nov 02 21:19:10.243333 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:10.243309 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:19:10.251275 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:19:10.251250 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79af2810_e0dc_4385_8714_2fd71edead77.slice/crio-7cc78c3019c681674516969092b35e09c4cea4585f32f272c4c6acfa3b2fef42 WatchSource:0}: Error finding container 7cc78c3019c681674516969092b35e09c4cea4585f32f272c4c6acfa3b2fef42: Status 404 returned error can't find the container with id 7cc78c3019c681674516969092b35e09c4cea4585f32f272c4c6acfa3b2fef42 Nov 02 21:19:10.718703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:10.718663 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" event={"ID":"b166645b-9667-4588-91bc-40bad8d01fd1","Type":"ContainerStarted","Data":"3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995"} Nov 02 21:19:10.719122 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:10.718734 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:10.719855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:10.719820 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" event={"ID":"79af2810-e0dc-4385-8714-2fd71edead77","Type":"ContainerStarted","Data":"7cc78c3019c681674516969092b35e09c4cea4585f32f272c4c6acfa3b2fef42"} Nov 02 21:19:10.720841 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:10.720820 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" event={"ID":"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67","Type":"ContainerStarted","Data":"ea94499f1dca816257770cf3645fa0cf44057106af4d82de6fffaef2d556bb9b"} Nov 02 21:19:10.825273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:10.825225 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" podStartSLOduration=1.5724199909999999 podStartE2EDuration="4.825211314s" totalImagesPullingTime="3.252791323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:10.82464199 +0000 UTC m=+376.800939776" watchObservedRunningTime="2026-11-02 21:19:10.825211314 +0000 UTC m=+376.801509088" Nov 02 21:19:11.727716 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:11.727680 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" event={"ID":"79af2810-e0dc-4385-8714-2fd71edead77","Type":"ContainerStarted","Data":"a0b691edff032cd2a765d6402a639ccea53e30412d95191fcc96ac4502ec12fc"} Nov 02 21:19:11.728073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:11.727811 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:11.833672 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:11.833613 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" podStartSLOduration=1.427095757 podStartE2EDuration="2.833598778s" totalImagesPullingTime="1.406503021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:11.832811404 +0000 UTC m=+377.809109190" watchObservedRunningTime="2026-11-02 21:19:11.833598778 +0000 UTC m=+377.809896554" Nov 02 21:19:14.742479 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:14.742442 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" event={"ID":"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67","Type":"ContainerStarted","Data":"61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48"} Nov 02 21:19:14.846280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:14.846213 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" podStartSLOduration=1.982925319 podStartE2EDuration="5.846193006s" totalImagesPullingTime="3.863267687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:14.845173582 +0000 UTC m=+380.821471357" watchObservedRunningTime="2026-11-02 21:19:14.846193006 +0000 UTC m=+380.822490782" Nov 02 21:19:15.530554 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:19:15.530500 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:19:21.729292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:21.729259 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:22.732401 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:22.732368 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zw95m" Nov 02 21:19:22.860269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:22.860228 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll8ss"] Nov 02 21:19:22.860458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:22.860404 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" podUID="b166645b-9667-4588-91bc-40bad8d01fd1" containerName="limitador" containerID="cri-o://3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995" gracePeriod=30 Nov 02 21:19:23.405558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.405534 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:23.537775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.536733 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b166645b-9667-4588-91bc-40bad8d01fd1-config-file\" (UniqueName: \"kubernetes.io/configmap/b166645b-9667-4588-91bc-40bad8d01fd1-config-file\") pod \"b166645b-9667-4588-91bc-40bad8d01fd1\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " Nov 02 21:19:23.537775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.536843 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b166645b-9667-4588-91bc-40bad8d01fd1-kube-api-access-g5gv4\" (UniqueName: \"kubernetes.io/projected/b166645b-9667-4588-91bc-40bad8d01fd1-kube-api-access-g5gv4\") pod \"b166645b-9667-4588-91bc-40bad8d01fd1\" (UID: \"b166645b-9667-4588-91bc-40bad8d01fd1\") " Nov 02 21:19:23.537775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.537689 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b166645b-9667-4588-91bc-40bad8d01fd1-config-file" pod "b166645b-9667-4588-91bc-40bad8d01fd1" (UID: "b166645b-9667-4588-91bc-40bad8d01fd1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:19:23.539921 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.539871 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b166645b-9667-4588-91bc-40bad8d01fd1-kube-api-access-g5gv4" pod "b166645b-9667-4588-91bc-40bad8d01fd1" (UID: "b166645b-9667-4588-91bc-40bad8d01fd1"). InnerVolumeSpecName "kube-api-access-g5gv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:19:23.638012 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.637972 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gv4\" (UniqueName: \"kubernetes.io/projected/b166645b-9667-4588-91bc-40bad8d01fd1-kube-api-access-g5gv4\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:23.638012 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.638002 4510 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b166645b-9667-4588-91bc-40bad8d01fd1-config-file\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:19:23.780789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.780753 4510 generic.go:356] "Generic (PLEG): container finished" podID="b166645b-9667-4588-91bc-40bad8d01fd1" containerID="3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995" exitCode=0 Nov 02 21:19:23.781170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.780802 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" Nov 02 21:19:23.781170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.780825 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" event={"ID":"b166645b-9667-4588-91bc-40bad8d01fd1","Type":"ContainerDied","Data":"3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995"} Nov 02 21:19:23.781170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.780870 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-ll8ss" event={"ID":"b166645b-9667-4588-91bc-40bad8d01fd1","Type":"ContainerDied","Data":"1f46739b3c8c974c9300ddbd4a6dba3e14d98e4ae9929b28889b373d7d3045f3"} Nov 02 21:19:23.781170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.780892 4510 scope.go:122] "RemoveContainer" containerID="3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995" Nov 02 21:19:23.790498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.790481 4510 scope.go:122] "RemoveContainer" containerID="3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995" Nov 02 21:19:23.790779 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:19:23.790759 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995\": container with ID starting with 3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995 not found: ID does not exist" containerID="3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995" Nov 02 21:19:23.790826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.790788 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995"} err="failed to get container status \"3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995\": rpc error: code = NotFound desc = could not find container \"3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995\": container with ID starting with 3f8f8eaad4ecb6a2252d13ae3eb23db89be23b0a15384d37a91299ae3c5f3995 not found: ID does not exist" Nov 02 21:19:23.941936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.941905 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll8ss"] Nov 02 21:19:23.988357 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:23.988324 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-ll8ss"] Nov 02 21:19:24.151380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:24.151349 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b166645b-9667-4588-91bc-40bad8d01fd1" path="/var/lib/kubelet/pods/b166645b-9667-4588-91bc-40bad8d01fd1/volumes" Nov 02 21:19:27.869042 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.869009 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd"] Nov 02 21:19:27.869430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.869367 4510 cpu_manager.go:414] "containerMap: removing container" podUID="b166645b-9667-4588-91bc-40bad8d01fd1" containerName="limitador" Nov 02 21:19:27.869430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.869378 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b166645b-9667-4588-91bc-40bad8d01fd1" containerName="limitador" Nov 02 21:19:27.869500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.869455 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b166645b-9667-4588-91bc-40bad8d01fd1" containerName="limitador" Nov 02 21:19:27.873639 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.873619 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:27.925602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.925573 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd"] Nov 02 21:19:27.968284 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.968254 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/beb13fd7-9ea5-4d0a-af9b-501b200280a4-data\") pod \"postgres-7cdbf6dc44-49lsd\" (UID: \"beb13fd7-9ea5-4d0a-af9b-501b200280a4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:27.968403 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.968331 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvvf\" (UniqueName: \"kubernetes.io/projected/beb13fd7-9ea5-4d0a-af9b-501b200280a4-kube-api-access-zgvvf\") pod \"postgres-7cdbf6dc44-49lsd\" (UID: \"beb13fd7-9ea5-4d0a-af9b-501b200280a4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:27.969593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.969578 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:19:27.969636 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.969595 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:19:27.979591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.979492 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 21:19:27.979591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:27.979514 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wvggz\"" Nov 02 21:19:28.068860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:28.068834 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/beb13fd7-9ea5-4d0a-af9b-501b200280a4-data\") pod \"postgres-7cdbf6dc44-49lsd\" (UID: \"beb13fd7-9ea5-4d0a-af9b-501b200280a4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:28.068963 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:28.068877 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvvf\" (UniqueName: \"kubernetes.io/projected/beb13fd7-9ea5-4d0a-af9b-501b200280a4-kube-api-access-zgvvf\") pod \"postgres-7cdbf6dc44-49lsd\" (UID: \"beb13fd7-9ea5-4d0a-af9b-501b200280a4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:28.069182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:28.069165 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/beb13fd7-9ea5-4d0a-af9b-501b200280a4-data\") pod \"postgres-7cdbf6dc44-49lsd\" (UID: \"beb13fd7-9ea5-4d0a-af9b-501b200280a4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:28.121625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:28.121565 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvvf\" (UniqueName: \"kubernetes.io/projected/beb13fd7-9ea5-4d0a-af9b-501b200280a4-kube-api-access-zgvvf\") pod \"postgres-7cdbf6dc44-49lsd\" (UID: \"beb13fd7-9ea5-4d0a-af9b-501b200280a4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:28.185429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:28.185400 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:28.553218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:28.553135 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd"] Nov 02 21:19:28.560850 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:19:28.560823 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb13fd7_9ea5_4d0a_af9b_501b200280a4.slice/crio-11aa0455e2b9784cda5fc1734e45cb1d76b82a167dfe1dabb897bfbbfbf2a3c8 WatchSource:0}: Error finding container 11aa0455e2b9784cda5fc1734e45cb1d76b82a167dfe1dabb897bfbbfbf2a3c8: Status 404 returned error can't find the container with id 11aa0455e2b9784cda5fc1734e45cb1d76b82a167dfe1dabb897bfbbfbf2a3c8 Nov 02 21:19:28.803546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:28.803462 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" event={"ID":"beb13fd7-9ea5-4d0a-af9b-501b200280a4","Type":"ContainerStarted","Data":"11aa0455e2b9784cda5fc1734e45cb1d76b82a167dfe1dabb897bfbbfbf2a3c8"} Nov 02 21:19:36.847699 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:36.847665 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" event={"ID":"beb13fd7-9ea5-4d0a-af9b-501b200280a4","Type":"ContainerStarted","Data":"07d70379022bfaa7e9cf830318ec4423a1f7b88dcf71f2b8bd9f11cf92a3515b"} Nov 02 21:19:36.848119 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:36.847749 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:36.951998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:36.951942 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" podStartSLOduration=2.460310014 podStartE2EDuration="9.951924576s" totalImagesPullingTime="7.491614562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:19:36.951476631 +0000 UTC m=+402.927774403" watchObservedRunningTime="2026-11-02 21:19:36.951924576 +0000 UTC m=+402.928222355" Nov 02 21:19:42.881369 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:42.881341 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-49lsd" Nov 02 21:19:45.530231 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:19:45.530197 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:19:57.188928 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.188894 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml"] Nov 02 21:19:57.196907 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.196883 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.245727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.245692 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml"] Nov 02 21:19:57.290494 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.290466 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 21:19:57.290670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.290573 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 21:19:57.290670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.290547 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 21:19:57.290670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.290467 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 21:19:57.290670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.290592 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-c4tkl\"" Nov 02 21:19:57.290830 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.290719 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 21:19:57.321057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.321030 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5fbd333-e69a-46ad-9059-0f9adc865b1e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.321182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.321064 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.321182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.321102 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.321262 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.321189 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjfm\" (UniqueName: \"kubernetes.io/projected/f5fbd333-e69a-46ad-9059-0f9adc865b1e-kube-api-access-rjjfm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.422281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.422251 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5fbd333-e69a-46ad-9059-0f9adc865b1e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.422445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.422286 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.422445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.422318 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.422445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.422356 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjfm\" (UniqueName: \"kubernetes.io/projected/f5fbd333-e69a-46ad-9059-0f9adc865b1e-kube-api-access-rjjfm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.422445 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:19:57.422390 4510 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 21:19:57.422685 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:19:57.422481 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-proxy-source podName:f5fbd333-e69a-46ad-9059-0f9adc865b1e nodeName:}" failed. No retries permitted until 2026-11-02 21:19:57.922460429 +0000 UTC m=+423.898758202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-r5kml" (UID: "f5fbd333-e69a-46ad-9059-0f9adc865b1e") : configmap "usage-tenancy-proxy-source" not found Nov 02 21:19:57.423059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.423035 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.424716 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.424700 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5fbd333-e69a-46ad-9059-0f9adc865b1e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.475474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.475416 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjfm\" (UniqueName: \"kubernetes.io/projected/f5fbd333-e69a-46ad-9059-0f9adc865b1e-kube-api-access-rjjfm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.927000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.926966 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:57.927801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:57.927780 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f5fbd333-e69a-46ad-9059-0f9adc865b1e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-r5kml\" (UID: \"f5fbd333-e69a-46ad-9059-0f9adc865b1e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:58.109625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:58.109582 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:19:58.476747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:58.476707 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml"] Nov 02 21:19:58.485165 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:19:58.485136 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fbd333_e69a_46ad_9059_0f9adc865b1e.slice/crio-71e05eb92229340d60b5d39fcb57c99c8d8b98c048f6188822b4a20b19220e4d WatchSource:0}: Error finding container 71e05eb92229340d60b5d39fcb57c99c8d8b98c048f6188822b4a20b19220e4d: Status 404 returned error can't find the container with id 71e05eb92229340d60b5d39fcb57c99c8d8b98c048f6188822b4a20b19220e4d Nov 02 21:19:58.943129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:19:58.943097 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" event={"ID":"f5fbd333-e69a-46ad-9059-0f9adc865b1e","Type":"ContainerStarted","Data":"71e05eb92229340d60b5d39fcb57c99c8d8b98c048f6188822b4a20b19220e4d"} Nov 02 21:20:15.529472 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:20:15.529438 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:20:16.032839 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:16.032806 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" event={"ID":"f5fbd333-e69a-46ad-9059-0f9adc865b1e","Type":"ContainerStarted","Data":"7b04e295b23273b990f70989164c17bd64d634c670b785614ab54414c689090d"} Nov 02 21:20:16.033020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:16.032960 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:20:16.140412 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:16.140360 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" podStartSLOduration=2.013393882 podStartE2EDuration="19.140344159s" totalImagesPullingTime="17.126950277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:16.138953397 +0000 UTC m=+442.115251169" watchObservedRunningTime="2026-11-02 21:20:16.140344159 +0000 UTC m=+442.116641935" Nov 02 21:20:20.959790 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:20.959756 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sxccp"] Nov 02 21:20:20.963079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:20.963060 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.055679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.055649 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84390d4-96bf-4c9d-90c6-8755afe0bc6b-tls-certs\") pod \"model-serving-api-75bd89fdfc-sxccp\" (UID: \"e84390d4-96bf-4c9d-90c6-8755afe0bc6b\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.055809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.055691 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5894k\" (UniqueName: \"kubernetes.io/projected/e84390d4-96bf-4c9d-90c6-8755afe0bc6b-kube-api-access-5894k\") pod \"model-serving-api-75bd89fdfc-sxccp\" (UID: \"e84390d4-96bf-4c9d-90c6-8755afe0bc6b\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.058010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.057990 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 21:20:21.058228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.058214 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-7vzdb\"" Nov 02 21:20:21.059703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.059685 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sxccp"] Nov 02 21:20:21.107304 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.107273 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5z6g"] Nov 02 21:20:21.110937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.110921 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.140773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.140750 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5z6g"] Nov 02 21:20:21.156710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.156689 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84390d4-96bf-4c9d-90c6-8755afe0bc6b-tls-certs\") pod \"model-serving-api-75bd89fdfc-sxccp\" (UID: \"e84390d4-96bf-4c9d-90c6-8755afe0bc6b\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.156831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.156736 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5894k\" (UniqueName: \"kubernetes.io/projected/e84390d4-96bf-4c9d-90c6-8755afe0bc6b-kube-api-access-5894k\") pod \"model-serving-api-75bd89fdfc-sxccp\" (UID: \"e84390d4-96bf-4c9d-90c6-8755afe0bc6b\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.156831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.156765 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j8fv\" (UniqueName: \"kubernetes.io/projected/ad711180-966a-47e7-b085-f0dd00cedbd5-kube-api-access-9j8fv\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.156920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.156891 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad711180-966a-47e7-b085-f0dd00cedbd5-cert\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.156969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.156920 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad711180-966a-47e7-b085-f0dd00cedbd5-metrics-certs\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.159264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.159238 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84390d4-96bf-4c9d-90c6-8755afe0bc6b-tls-certs\") pod \"model-serving-api-75bd89fdfc-sxccp\" (UID: \"e84390d4-96bf-4c9d-90c6-8755afe0bc6b\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.159550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.159515 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 02 21:20:21.159629 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.159582 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9wrgp\"" Nov 02 21:20:21.159674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.159666 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 02 21:20:21.209210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.209189 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5894k\" (UniqueName: \"kubernetes.io/projected/e84390d4-96bf-4c9d-90c6-8755afe0bc6b-kube-api-access-5894k\") pod \"model-serving-api-75bd89fdfc-sxccp\" (UID: \"e84390d4-96bf-4c9d-90c6-8755afe0bc6b\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.257921 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.257866 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j8fv\" (UniqueName: \"kubernetes.io/projected/ad711180-966a-47e7-b085-f0dd00cedbd5-kube-api-access-9j8fv\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.258018 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.257924 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad711180-966a-47e7-b085-f0dd00cedbd5-cert\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.258018 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.257943 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad711180-966a-47e7-b085-f0dd00cedbd5-metrics-certs\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.260103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.260082 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad711180-966a-47e7-b085-f0dd00cedbd5-metrics-certs\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.260309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.260289 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad711180-966a-47e7-b085-f0dd00cedbd5-cert\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.276059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.276037 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:21.311014 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.310977 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j8fv\" (UniqueName: \"kubernetes.io/projected/ad711180-966a-47e7-b085-f0dd00cedbd5-kube-api-access-9j8fv\") pod \"odh-model-controller-769875db56-n5z6g\" (UID: \"ad711180-966a-47e7-b085-f0dd00cedbd5\") " pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.423441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.423412 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:21.641043 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.641013 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sxccp"] Nov 02 21:20:21.648743 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:20:21.648718 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84390d4_96bf_4c9d_90c6_8755afe0bc6b.slice/crio-bb77ecbb1b040c62ce7c0f212d78cd37c652bf392e27a7f0959c65549a9118da WatchSource:0}: Error finding container bb77ecbb1b040c62ce7c0f212d78cd37c652bf392e27a7f0959c65549a9118da: Status 404 returned error can't find the container with id bb77ecbb1b040c62ce7c0f212d78cd37c652bf392e27a7f0959c65549a9118da Nov 02 21:20:21.790977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:21.790945 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-n5z6g"] Nov 02 21:20:21.798680 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:20:21.798653 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad711180_966a_47e7_b085_f0dd00cedbd5.slice/crio-1a8d7afe966f29b8786b941d3ff060e373fc1124edb8658ec7b716011630ed72 WatchSource:0}: Error finding container 1a8d7afe966f29b8786b941d3ff060e373fc1124edb8658ec7b716011630ed72: Status 404 returned error can't find the container with id 1a8d7afe966f29b8786b941d3ff060e373fc1124edb8658ec7b716011630ed72 Nov 02 21:20:22.059384 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:22.059297 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n5z6g" event={"ID":"ad711180-966a-47e7-b085-f0dd00cedbd5","Type":"ContainerStarted","Data":"1a8d7afe966f29b8786b941d3ff060e373fc1124edb8658ec7b716011630ed72"} Nov 02 21:20:22.060413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:22.060388 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" event={"ID":"e84390d4-96bf-4c9d-90c6-8755afe0bc6b","Type":"ContainerStarted","Data":"bb77ecbb1b040c62ce7c0f212d78cd37c652bf392e27a7f0959c65549a9118da"} Nov 02 21:20:25.081787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:25.081760 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" event={"ID":"e84390d4-96bf-4c9d-90c6-8755afe0bc6b","Type":"ContainerStarted","Data":"09fe292679d9ab246d703b9f11745d367aabdb6a7004a19e3bbffa92277411cc"} Nov 02 21:20:25.082121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:25.081870 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:25.187099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:25.186993 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" podStartSLOduration=2.696785938 podStartE2EDuration="5.186977203s" totalImagesPullingTime="2.490191265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:25.185892988 +0000 UTC m=+451.162190765" watchObservedRunningTime="2026-11-02 21:20:25.186977203 +0000 UTC m=+451.163274979" Nov 02 21:20:26.087646 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:26.087612 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-n5z6g" event={"ID":"ad711180-966a-47e7-b085-f0dd00cedbd5","Type":"ContainerStarted","Data":"c93868c5cd43ab5c980836322522229ccc335bc5f2106d7e6e9df878c638d705"} Nov 02 21:20:26.088115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:26.087838 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:26.195278 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:26.195222 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-n5z6g" podStartSLOduration=1.9373763259999999 podStartE2EDuration="5.195207469s" totalImagesPullingTime="3.257831143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:26.193139857 +0000 UTC m=+452.169437613" watchObservedRunningTime="2026-11-02 21:20:26.195207469 +0000 UTC m=+452.171505261" Nov 02 21:20:27.042245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:27.042206 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-r5kml" Nov 02 21:20:28.920281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:28.920251 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47"] Nov 02 21:20:28.924103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:28.924082 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.020665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.020630 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 21:20:29.020665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.020650 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:20:29.020665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.020650 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47p47\"" Nov 02 21:20:29.022301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.022285 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47"] Nov 02 21:20:29.026113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.026082 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pc95\" (UniqueName: \"kubernetes.io/projected/e198d705-b481-4134-97e6-e12b73601fdc-kube-api-access-6pc95\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.026215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.026152 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-maas-api-tls\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.026256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.026237 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-metrics-certs\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.026312 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.026299 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198d705-b481-4134-97e6-e12b73601fdc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.127625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.127588 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-metrics-certs\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.127774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.127656 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198d705-b481-4134-97e6-e12b73601fdc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.127774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.127701 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pc95\" (UniqueName: \"kubernetes.io/projected/e198d705-b481-4134-97e6-e12b73601fdc-kube-api-access-6pc95\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.127774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.127760 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-maas-api-tls\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.128451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.128423 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198d705-b481-4134-97e6-e12b73601fdc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.130172 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.130148 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-metrics-certs\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.130266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.130194 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-maas-api-tls\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.180674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.180617 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pc95\" (UniqueName: \"kubernetes.io/projected/e198d705-b481-4134-97e6-e12b73601fdc-kube-api-access-6pc95\") pod \"maas-api-6d669c7fcd-zdc47\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.236360 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.236337 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:29.621965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:29.621928 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47"] Nov 02 21:20:29.629692 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:20:29.629661 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode198d705_b481_4134_97e6_e12b73601fdc.slice/crio-4cbfd116e9f8929bcee4baecb1e945dee7084b3ce27652834dbaffae7571e43f WatchSource:0}: Error finding container 4cbfd116e9f8929bcee4baecb1e945dee7084b3ce27652834dbaffae7571e43f: Status 404 returned error can't find the container with id 4cbfd116e9f8929bcee4baecb1e945dee7084b3ce27652834dbaffae7571e43f Nov 02 21:20:30.105978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:30.105901 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" event={"ID":"e198d705-b481-4134-97e6-e12b73601fdc","Type":"ContainerStarted","Data":"4cbfd116e9f8929bcee4baecb1e945dee7084b3ce27652834dbaffae7571e43f"} Nov 02 21:20:33.122007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:33.121976 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" event={"ID":"e198d705-b481-4134-97e6-e12b73601fdc","Type":"ContainerStarted","Data":"f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3"} Nov 02 21:20:33.122367 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:33.122040 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:33.228483 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:33.228338 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" podStartSLOduration=2.006192197 podStartE2EDuration="5.228319868s" totalImagesPullingTime="3.222127671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:33.227398093 +0000 UTC m=+459.203695868" watchObservedRunningTime="2026-11-02 21:20:33.228319868 +0000 UTC m=+459.204617645" Nov 02 21:20:34.426815 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:34.426776 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:20:36.091445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:36.091415 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-sxccp" Nov 02 21:20:37.094363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:37.094335 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-n5z6g" Nov 02 21:20:39.130849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:39.130818 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:39.906333 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:39.906295 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9rnrv"] Nov 02 21:20:39.909999 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:39.909979 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" Nov 02 21:20:39.963463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:39.963405 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9rnrv"] Nov 02 21:20:40.032384 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:40.032354 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf28j\" (UniqueName: \"kubernetes.io/projected/b1b097a4-741e-4785-af27-e18f6e54b90f-kube-api-access-sf28j\") pod \"authorino-6f6944bcdd-9rnrv\" (UID: \"b1b097a4-741e-4785-af27-e18f6e54b90f\") " pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" Nov 02 21:20:40.133090 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:40.133061 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf28j\" (UniqueName: \"kubernetes.io/projected/b1b097a4-741e-4785-af27-e18f6e54b90f-kube-api-access-sf28j\") pod \"authorino-6f6944bcdd-9rnrv\" (UID: \"b1b097a4-741e-4785-af27-e18f6e54b90f\") " pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" Nov 02 21:20:40.186345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:40.186273 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf28j\" (UniqueName: \"kubernetes.io/projected/b1b097a4-741e-4785-af27-e18f6e54b90f-kube-api-access-sf28j\") pod \"authorino-6f6944bcdd-9rnrv\" (UID: \"b1b097a4-741e-4785-af27-e18f6e54b90f\") " pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" Nov 02 21:20:40.221087 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:40.221057 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" Nov 02 21:20:40.247314 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:40.247283 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9rnrv"] Nov 02 21:20:40.586759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:40.586688 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9rnrv"] Nov 02 21:20:40.594664 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:20:40.594634 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1b097a4_741e_4785_af27_e18f6e54b90f.slice/crio-d355b1b6e90485cb924cf7ddaa29d7d61b54b1d3b17d2cbf23ce56c107e7b907 WatchSource:0}: Error finding container d355b1b6e90485cb924cf7ddaa29d7d61b54b1d3b17d2cbf23ce56c107e7b907: Status 404 returned error can't find the container with id d355b1b6e90485cb924cf7ddaa29d7d61b54b1d3b17d2cbf23ce56c107e7b907 Nov 02 21:20:41.155914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:41.155875 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" event={"ID":"b1b097a4-741e-4785-af27-e18f6e54b90f","Type":"ContainerStarted","Data":"d355b1b6e90485cb924cf7ddaa29d7d61b54b1d3b17d2cbf23ce56c107e7b907"} Nov 02 21:20:43.165088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.165054 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" event={"ID":"b1b097a4-741e-4785-af27-e18f6e54b90f","Type":"ContainerStarted","Data":"b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628"} Nov 02 21:20:43.165514 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.165120 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" podUID="b1b097a4-741e-4785-af27-e18f6e54b90f" containerName="authorino" containerID="cri-o://b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628" gracePeriod=30 Nov 02 21:20:43.269330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.269271 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" podStartSLOduration=2.4781993509999998 podStartE2EDuration="4.269253605s" totalImagesPullingTime="1.791054254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:20:43.268217466 +0000 UTC m=+469.244515241" watchObservedRunningTime="2026-11-02 21:20:43.269253605 +0000 UTC m=+469.245551381" Nov 02 21:20:43.387091 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.386880 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xnb8h"] Nov 02 21:20:43.387420 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.387285 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" podUID="474b5b1a-9103-4ef9-bc9c-81e41b5cfa67" containerName="authorino" containerID="cri-o://61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48" gracePeriod=30 Nov 02 21:20:43.549338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.549316 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" Nov 02 21:20:43.665136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.665107 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b097a4-741e-4785-af27-e18f6e54b90f-kube-api-access-sf28j\" (UniqueName: \"kubernetes.io/projected/b1b097a4-741e-4785-af27-e18f6e54b90f-kube-api-access-sf28j\") pod \"b1b097a4-741e-4785-af27-e18f6e54b90f\" (UID: \"b1b097a4-741e-4785-af27-e18f6e54b90f\") " Nov 02 21:20:43.667464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.667440 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b097a4-741e-4785-af27-e18f6e54b90f-kube-api-access-sf28j" pod "b1b097a4-741e-4785-af27-e18f6e54b90f" (UID: "b1b097a4-741e-4785-af27-e18f6e54b90f"). InnerVolumeSpecName "kube-api-access-sf28j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:20:43.700505 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.700483 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" Nov 02 21:20:43.766708 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.766656 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67-kube-api-access-bbnvg\" (UniqueName: \"kubernetes.io/projected/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67-kube-api-access-bbnvg\") pod \"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67\" (UID: \"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67\") " Nov 02 21:20:43.766855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.766838 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf28j\" (UniqueName: \"kubernetes.io/projected/b1b097a4-741e-4785-af27-e18f6e54b90f-kube-api-access-sf28j\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:43.768531 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.768494 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67-kube-api-access-bbnvg" pod "474b5b1a-9103-4ef9-bc9c-81e41b5cfa67" (UID: "474b5b1a-9103-4ef9-bc9c-81e41b5cfa67"). InnerVolumeSpecName "kube-api-access-bbnvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:20:43.867791 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:43.867762 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbnvg\" (UniqueName: \"kubernetes.io/projected/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67-kube-api-access-bbnvg\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:44.170240 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.170213 4510 generic.go:356] "Generic (PLEG): container finished" podID="b1b097a4-741e-4785-af27-e18f6e54b90f" containerID="b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628" exitCode=0 Nov 02 21:20:44.170607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.170255 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" Nov 02 21:20:44.170607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.170290 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" event={"ID":"b1b097a4-741e-4785-af27-e18f6e54b90f","Type":"ContainerDied","Data":"b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628"} Nov 02 21:20:44.170607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.170323 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9rnrv" event={"ID":"b1b097a4-741e-4785-af27-e18f6e54b90f","Type":"ContainerDied","Data":"d355b1b6e90485cb924cf7ddaa29d7d61b54b1d3b17d2cbf23ce56c107e7b907"} Nov 02 21:20:44.170607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.170340 4510 scope.go:122] "RemoveContainer" containerID="b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628" Nov 02 21:20:44.171647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.171624 4510 generic.go:356] "Generic (PLEG): container finished" podID="474b5b1a-9103-4ef9-bc9c-81e41b5cfa67" containerID="61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48" exitCode=0 Nov 02 21:20:44.171725 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.171670 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" Nov 02 21:20:44.171778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.171720 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" event={"ID":"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67","Type":"ContainerDied","Data":"61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48"} Nov 02 21:20:44.171778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.171756 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xnb8h" event={"ID":"474b5b1a-9103-4ef9-bc9c-81e41b5cfa67","Type":"ContainerDied","Data":"ea94499f1dca816257770cf3645fa0cf44057106af4d82de6fffaef2d556bb9b"} Nov 02 21:20:44.178612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.178597 4510 scope.go:122] "RemoveContainer" containerID="b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628" Nov 02 21:20:44.178873 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:20:44.178853 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628\": container with ID starting with b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628 not found: ID does not exist" containerID="b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628" Nov 02 21:20:44.178932 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.178881 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628"} err="failed to get container status \"b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628\": rpc error: code = NotFound desc = could not find container \"b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628\": container with ID starting with b1a192c99e5f23cdc60ee259b485a0ef9b21266857daedaafad46c5b88299628 not found: ID does not exist" Nov 02 21:20:44.178932 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.178897 4510 scope.go:122] "RemoveContainer" containerID="61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48" Nov 02 21:20:44.186562 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.186546 4510 scope.go:122] "RemoveContainer" containerID="61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48" Nov 02 21:20:44.186802 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:20:44.186785 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48\": container with ID starting with 61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48 not found: ID does not exist" containerID="61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48" Nov 02 21:20:44.186866 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.186811 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48"} err="failed to get container status \"61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48\": rpc error: code = NotFound desc = could not find container \"61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48\": container with ID starting with 61527b60def2929d022ec880b55559ea77729af9055a4977b5e97a4fcca0ad48 not found: ID does not exist" Nov 02 21:20:44.320061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.320033 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9rnrv"] Nov 02 21:20:44.366040 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.366014 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9rnrv"] Nov 02 21:20:44.465834 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.465776 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xnb8h"] Nov 02 21:20:44.467721 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:44.467701 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xnb8h"] Nov 02 21:20:45.531548 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:20:45.531487 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:20:46.156949 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:46.156913 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474b5b1a-9103-4ef9-bc9c-81e41b5cfa67" path="/var/lib/kubelet/pods/474b5b1a-9103-4ef9-bc9c-81e41b5cfa67/volumes" Nov 02 21:20:46.157248 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:46.157235 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b097a4-741e-4785-af27-e18f6e54b90f" path="/var/lib/kubelet/pods/b1b097a4-741e-4785-af27-e18f6e54b90f/volumes" Nov 02 21:20:49.773394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:49.773361 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47"] Nov 02 21:20:49.773891 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:49.773566 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" podUID="e198d705-b481-4134-97e6-e12b73601fdc" containerName="maas-api" containerID="cri-o://f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3" gracePeriod=30 Nov 02 21:20:50.018318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.018290 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:50.122354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.122327 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-maas-api-tls\") pod \"e198d705-b481-4134-97e6-e12b73601fdc\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " Nov 02 21:20:50.122542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.122360 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e198d705-b481-4134-97e6-e12b73601fdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198d705-b481-4134-97e6-e12b73601fdc-openshift-service-ca\") pod \"e198d705-b481-4134-97e6-e12b73601fdc\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " Nov 02 21:20:50.122542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.122416 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e198d705-b481-4134-97e6-e12b73601fdc-kube-api-access-6pc95\" (UniqueName: \"kubernetes.io/projected/e198d705-b481-4134-97e6-e12b73601fdc-kube-api-access-6pc95\") pod \"e198d705-b481-4134-97e6-e12b73601fdc\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " Nov 02 21:20:50.122542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.122465 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-metrics-certs\") pod \"e198d705-b481-4134-97e6-e12b73601fdc\" (UID: \"e198d705-b481-4134-97e6-e12b73601fdc\") " Nov 02 21:20:50.122934 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.122894 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e198d705-b481-4134-97e6-e12b73601fdc-openshift-service-ca" pod "e198d705-b481-4134-97e6-e12b73601fdc" (UID: "e198d705-b481-4134-97e6-e12b73601fdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:20:50.124856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.124727 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-maas-api-tls" pod "e198d705-b481-4134-97e6-e12b73601fdc" (UID: "e198d705-b481-4134-97e6-e12b73601fdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:20:50.124856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.124846 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e198d705-b481-4134-97e6-e12b73601fdc-kube-api-access-6pc95" pod "e198d705-b481-4134-97e6-e12b73601fdc" (UID: "e198d705-b481-4134-97e6-e12b73601fdc"). InnerVolumeSpecName "kube-api-access-6pc95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:20:50.125002 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.124929 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-metrics-certs" pod "e198d705-b481-4134-97e6-e12b73601fdc" (UID: "e198d705-b481-4134-97e6-e12b73601fdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:20:50.199848 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.199817 4510 generic.go:356] "Generic (PLEG): container finished" podID="e198d705-b481-4134-97e6-e12b73601fdc" containerID="f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3" exitCode=0 Nov 02 21:20:50.199967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.199861 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" Nov 02 21:20:50.199967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.199900 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" event={"ID":"e198d705-b481-4134-97e6-e12b73601fdc","Type":"ContainerDied","Data":"f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3"} Nov 02 21:20:50.199967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.199948 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47" event={"ID":"e198d705-b481-4134-97e6-e12b73601fdc","Type":"ContainerDied","Data":"4cbfd116e9f8929bcee4baecb1e945dee7084b3ce27652834dbaffae7571e43f"} Nov 02 21:20:50.200081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.199970 4510 scope.go:122] "RemoveContainer" containerID="f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3" Nov 02 21:20:50.208318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.208301 4510 scope.go:122] "RemoveContainer" containerID="f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3" Nov 02 21:20:50.208575 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:20:50.208558 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3\": container with ID starting with f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3 not found: ID does not exist" containerID="f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3" Nov 02 21:20:50.208625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.208584 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3"} err="failed to get container status \"f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3\": rpc error: code = NotFound desc = could not find container \"f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3\": container with ID starting with f7b15f47cbcfb4cc70dfb90a6c69c144edd091a6fc4599b6fb7a005d9b75d9a3 not found: ID does not exist" Nov 02 21:20:50.224044 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.224023 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:50.224044 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.224044 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198d705-b481-4134-97e6-e12b73601fdc-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:50.224155 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.224053 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pc95\" (UniqueName: \"kubernetes.io/projected/e198d705-b481-4134-97e6-e12b73601fdc-kube-api-access-6pc95\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:50.224155 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.224062 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e198d705-b481-4134-97e6-e12b73601fdc-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:20:50.351414 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.351386 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47"] Nov 02 21:20:50.356744 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:50.356722 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-zdc47"] Nov 02 21:20:52.151967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:20:52.151931 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e198d705-b481-4134-97e6-e12b73601fdc" path="/var/lib/kubelet/pods/e198d705-b481-4134-97e6-e12b73601fdc/volumes" Nov 02 21:21:15.529823 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:21:15.529741 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:21:45.530231 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:21:45.530196 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:22:15.529938 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:22:15.529838 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:22:36.801026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.800946 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj"] Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801313 4510 cpu_manager.go:414] "containerMap: removing container" podUID="b1b097a4-741e-4785-af27-e18f6e54b90f" containerName="authorino" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801324 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b097a4-741e-4785-af27-e18f6e54b90f" containerName="authorino" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801340 4510 cpu_manager.go:414] "containerMap: removing container" podUID="e198d705-b481-4134-97e6-e12b73601fdc" containerName="maas-api" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801345 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e198d705-b481-4134-97e6-e12b73601fdc" containerName="maas-api" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801367 4510 cpu_manager.go:414] "containerMap: removing container" podUID="474b5b1a-9103-4ef9-bc9c-81e41b5cfa67" containerName="authorino" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801373 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474b5b1a-9103-4ef9-bc9c-81e41b5cfa67" containerName="authorino" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801432 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b097a4-741e-4785-af27-e18f6e54b90f" containerName="authorino" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801442 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e198d705-b481-4134-97e6-e12b73601fdc" containerName="maas-api" Nov 02 21:22:36.801484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.801451 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="474b5b1a-9103-4ef9-bc9c-81e41b5cfa67" containerName="authorino" Nov 02 21:22:36.804823 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.804805 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:36.853835 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.853803 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 21:22:36.899603 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.899568 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 21:22:36.899768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.899639 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 21:22:36.899768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.899649 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8m4js\"" Nov 02 21:22:36.901499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.901482 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj"] Nov 02 21:22:36.927890 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.927845 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de15bfd8-7fcb-486e-a603-8c4d513dadc9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:36.928010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.927910 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:36.928054 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.928014 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:36.928089 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.928072 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq42\" (UniqueName: \"kubernetes.io/projected/de15bfd8-7fcb-486e-a603-8c4d513dadc9-kube-api-access-bnq42\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:36.928145 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.928132 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:36.928179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:36.928161 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.028820 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.028786 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.028973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.028833 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.028973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.028885 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq42\" (UniqueName: \"kubernetes.io/projected/de15bfd8-7fcb-486e-a603-8c4d513dadc9-kube-api-access-bnq42\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.028973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.028927 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.028973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.028952 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.029187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.029018 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de15bfd8-7fcb-486e-a603-8c4d513dadc9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.029298 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.029273 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.029358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.029310 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.029358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.029332 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.031101 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.031082 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de15bfd8-7fcb-486e-a603-8c4d513dadc9-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.031437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.031417 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de15bfd8-7fcb-486e-a603-8c4d513dadc9-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.085405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.085348 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq42\" (UniqueName: \"kubernetes.io/projected/de15bfd8-7fcb-486e-a603-8c4d513dadc9-kube-api-access-bnq42\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-cmhbj\" (UID: \"de15bfd8-7fcb-486e-a603-8c4d513dadc9\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.118062 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.118040 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:37.486878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.486846 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj"] Nov 02 21:22:37.495087 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:22:37.495061 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde15bfd8_7fcb_486e_a603_8c4d513dadc9.slice/crio-10aa7dcc9ff136b8c35ae88bdc47380ee4119fdee9e0040a5d882e98e1b77f3e WatchSource:0}: Error finding container 10aa7dcc9ff136b8c35ae88bdc47380ee4119fdee9e0040a5d882e98e1b77f3e: Status 404 returned error can't find the container with id 10aa7dcc9ff136b8c35ae88bdc47380ee4119fdee9e0040a5d882e98e1b77f3e Nov 02 21:22:37.652248 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:37.652214 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" event={"ID":"de15bfd8-7fcb-486e-a603-8c4d513dadc9","Type":"ContainerStarted","Data":"10aa7dcc9ff136b8c35ae88bdc47380ee4119fdee9e0040a5d882e98e1b77f3e"} Nov 02 21:22:39.664078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:39.664044 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" event={"ID":"de15bfd8-7fcb-486e-a603-8c4d513dadc9","Type":"ContainerStarted","Data":"0a9abf9ed853574ab4113db153e8deefcae24dd6b762e711dfb4a5df5257f032"} Nov 02 21:22:39.770112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:39.770063 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" podStartSLOduration=2.075947066 podStartE2EDuration="3.770050426s" totalImagesPullingTime="1.69410336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:39.769095716 +0000 UTC m=+585.745393504" watchObservedRunningTime="2026-11-02 21:22:39.770050426 +0000 UTC m=+585.746348217" Nov 02 21:22:40.118499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:40.118464 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:40.472074 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:40.471994 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:22:41.625474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.625435 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6"] Nov 02 21:22:41.629347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.629327 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.679257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.679223 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1fe9ee-732a-4438-8a6d-a6353430059a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.679429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.679285 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.679429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.679356 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.679563 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.679464 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.679563 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.679505 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.679654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.679619 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9jh\" (UniqueName: \"kubernetes.io/projected/ab1fe9ee-732a-4438-8a6d-a6353430059a-kube-api-access-jn9jh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.723736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.723707 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 21:22:41.725295 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.725275 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6"] Nov 02 21:22:41.780036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780011 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780041 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780061 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9jh\" (UniqueName: \"kubernetes.io/projected/ab1fe9ee-732a-4438-8a6d-a6353430059a-kube-api-access-jn9jh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780114 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1fe9ee-732a-4438-8a6d-a6353430059a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780151 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780412 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780173 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780490 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780464 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780637 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.780784 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.780764 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.782405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.782388 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab1fe9ee-732a-4438-8a6d-a6353430059a-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.782679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.782661 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab1fe9ee-732a-4438-8a6d-a6353430059a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.832124 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.832099 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9jh\" (UniqueName: \"kubernetes.io/projected/ab1fe9ee-732a-4438-8a6d-a6353430059a-kube-api-access-jn9jh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-w4wp6\" (UID: \"ab1fe9ee-732a-4438-8a6d-a6353430059a\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:41.941761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:41.941676 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:42.308092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:42.308011 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6"] Nov 02 21:22:42.315432 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:22:42.315400 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1fe9ee_732a_4438_8a6d_a6353430059a.slice/crio-0afbb7da50e3d03b4160a45da60a4b62c642ec1dd94f2b89c1e74c2b58452c41 WatchSource:0}: Error finding container 0afbb7da50e3d03b4160a45da60a4b62c642ec1dd94f2b89c1e74c2b58452c41: Status 404 returned error can't find the container with id 0afbb7da50e3d03b4160a45da60a4b62c642ec1dd94f2b89c1e74c2b58452c41 Nov 02 21:22:42.679643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:42.679608 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" event={"ID":"ab1fe9ee-732a-4438-8a6d-a6353430059a","Type":"ContainerStarted","Data":"0afbb7da50e3d03b4160a45da60a4b62c642ec1dd94f2b89c1e74c2b58452c41"} Nov 02 21:22:43.685436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:43.685398 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" event={"ID":"ab1fe9ee-732a-4438-8a6d-a6353430059a","Type":"ContainerStarted","Data":"028d3f271fc606d47bf86cf42c85a2129b024339efd6b7b91ea7835d452eade9"} Nov 02 21:22:43.792249 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:43.792195 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" podStartSLOduration=2.166094004 podStartE2EDuration="2.792182501s" totalImagesPullingTime="626.088497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:43.791513854 +0000 UTC m=+589.767811629" watchObservedRunningTime="2026-11-02 21:22:43.792182501 +0000 UTC m=+589.768480276" Nov 02 21:22:43.942172 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:43.942099 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:45.530281 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:22:45.530245 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:22:45.601417 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.601385 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd"] Nov 02 21:22:45.605068 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.605049 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.699234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.699207 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 21:22:45.700905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.700882 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd"] Nov 02 21:22:45.718459 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.718427 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.718637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.718615 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.718690 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.718650 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.718690 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.718677 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwx26\" (UniqueName: \"kubernetes.io/projected/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-kube-api-access-fwx26\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.718819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.718798 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.718854 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.718838 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.819474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819414 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.819474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819443 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.819474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819461 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwx26\" (UniqueName: \"kubernetes.io/projected/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-kube-api-access-fwx26\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.819715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819535 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.819715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819563 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.819715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819588 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.820002 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819920 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.820002 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.819957 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.820002 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.820000 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.821773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.821752 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.822036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.822019 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.874780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.874753 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwx26\" (UniqueName: \"kubernetes.io/projected/38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0-kube-api-access-fwx26\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd\" (UID: \"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:45.917763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:45.917740 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:46.290303 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:46.290270 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd"] Nov 02 21:22:46.298034 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:22:46.298003 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cb3dd9_ad4b_47db_a1cb_0dd862d11bf0.slice/crio-3a23f2b5630291030fac81d1fa4cea4f575f6348ae9b085ed29ec122f65d9858 WatchSource:0}: Error finding container 3a23f2b5630291030fac81d1fa4cea4f575f6348ae9b085ed29ec122f65d9858: Status 404 returned error can't find the container with id 3a23f2b5630291030fac81d1fa4cea4f575f6348ae9b085ed29ec122f65d9858 Nov 02 21:22:46.699714 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:46.699678 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" event={"ID":"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0","Type":"ContainerStarted","Data":"3a23f2b5630291030fac81d1fa4cea4f575f6348ae9b085ed29ec122f65d9858"} Nov 02 21:22:47.118215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:47.118187 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:47.130162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:47.130137 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:47.706373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:47.706336 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" event={"ID":"38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0","Type":"ContainerStarted","Data":"9e94d8bdd0ad41049546963afe0c1f5bd0bbe1510a8abb7e74c41ab353f61cfb"} Nov 02 21:22:47.717445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:47.717421 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-cmhbj" Nov 02 21:22:47.812952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:47.812893 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" podStartSLOduration=2.179810126 podStartE2EDuration="2.812879468s" totalImagesPullingTime="633.069342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:47.811682848 +0000 UTC m=+593.787980633" watchObservedRunningTime="2026-11-02 21:22:47.812879468 +0000 UTC m=+593.789177242" Nov 02 21:22:47.917975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:47.917900 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:49.196162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.196121 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tpgnk"] Nov 02 21:22:49.199709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.199689 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.242318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.242290 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tpgnk"] Nov 02 21:22:49.250410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.250382 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-home\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.250553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.250413 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.250553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.250429 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rb47\" (UniqueName: \"kubernetes.io/projected/c42cb1c5-171a-4f0b-9607-b16a085ee542-kube-api-access-6rb47\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.250674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.250641 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42cb1c5-171a-4f0b-9607-b16a085ee542-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.250735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.250697 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.250788 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.250753 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.275171 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.275147 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 21:22:49.351666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.351637 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.351793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.351684 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-home\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.351793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.351701 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.351793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.351716 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rb47\" (UniqueName: \"kubernetes.io/projected/c42cb1c5-171a-4f0b-9607-b16a085ee542-kube-api-access-6rb47\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.351793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.351764 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42cb1c5-171a-4f0b-9607-b16a085ee542-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.351953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.351805 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.352079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.352057 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.352117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.352064 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-home\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.352158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.352130 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.353999 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.353976 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42cb1c5-171a-4f0b-9607-b16a085ee542-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.354399 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.354377 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42cb1c5-171a-4f0b-9607-b16a085ee542-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.404623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.404590 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rb47\" (UniqueName: \"kubernetes.io/projected/c42cb1c5-171a-4f0b-9607-b16a085ee542-kube-api-access-6rb47\") pod \"echo-simulator-kserve-64b6f8f96c-tpgnk\" (UID: \"c42cb1c5-171a-4f0b-9607-b16a085ee542\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.512845 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.512750 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:49.884996 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:49.884963 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tpgnk"] Nov 02 21:22:49.892725 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:22:49.892692 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42cb1c5_171a_4f0b_9607_b16a085ee542.slice/crio-0300f1d28215a48ba534a31d73c618e4a569b56ea5ca20dfb38d86c5d0d9d121 WatchSource:0}: Error finding container 0300f1d28215a48ba534a31d73c618e4a569b56ea5ca20dfb38d86c5d0d9d121: Status 404 returned error can't find the container with id 0300f1d28215a48ba534a31d73c618e4a569b56ea5ca20dfb38d86c5d0d9d121 Nov 02 21:22:50.726758 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:50.726719 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" event={"ID":"c42cb1c5-171a-4f0b-9607-b16a085ee542","Type":"ContainerStarted","Data":"0300f1d28215a48ba534a31d73c618e4a569b56ea5ca20dfb38d86c5d0d9d121"} Nov 02 21:22:51.760762 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:51.760728 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:22:51.942381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:51.942352 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:51.954514 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:51.954485 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:52.741488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:52.741395 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" event={"ID":"c42cb1c5-171a-4f0b-9607-b16a085ee542","Type":"ContainerStarted","Data":"9a51faeb21ae20a8781736169c6227ae5bce5d43d2cae59ab5cf6a0b3e65078b"} Nov 02 21:22:52.752746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:52.752718 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-w4wp6" Nov 02 21:22:52.848785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:52.848738 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" podStartSLOduration=1.330688694 podStartE2EDuration="3.848724823s" totalImagesPullingTime="2.518036129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:52.846970229 +0000 UTC m=+598.823268004" watchObservedRunningTime="2026-11-02 21:22:52.848724823 +0000 UTC m=+598.825022598" Nov 02 21:22:53.212826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.212797 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm"] Nov 02 21:22:53.216470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.216451 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.264861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.264838 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 21:22:53.293153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.293124 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.293267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.293166 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.293267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.293245 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94522fad-8716-4425-85ed-dba0714e23ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.293363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.293342 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.293432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.293416 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.293485 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.293461 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2psp\" (UniqueName: \"kubernetes.io/projected/94522fad-8716-4425-85ed-dba0714e23ad-kube-api-access-f2psp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.312451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.312430 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm"] Nov 02 21:22:53.394349 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394326 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94522fad-8716-4425-85ed-dba0714e23ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394361 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394387 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394423 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2psp\" (UniqueName: \"kubernetes.io/projected/94522fad-8716-4425-85ed-dba0714e23ad-kube-api-access-f2psp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394459 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394498 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394877 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394839 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394879 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.394950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.394942 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.396694 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.396673 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94522fad-8716-4425-85ed-dba0714e23ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.396867 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.396849 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94522fad-8716-4425-85ed-dba0714e23ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.447192 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.447171 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2psp\" (UniqueName: \"kubernetes.io/projected/94522fad-8716-4425-85ed-dba0714e23ad-kube-api-access-f2psp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-svskm\" (UID: \"94522fad-8716-4425-85ed-dba0714e23ad\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.512908 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.512855 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:53.529058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.529042 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:53.895493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:53.895465 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm"] Nov 02 21:22:53.902738 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:22:53.902709 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94522fad_8716_4425_85ed_dba0714e23ad.slice/crio-62c9679032e1ef110df0bb3ef17bd72b4e93a4b94bb23044b6d1cf7707059018 WatchSource:0}: Error finding container 62c9679032e1ef110df0bb3ef17bd72b4e93a4b94bb23044b6d1cf7707059018: Status 404 returned error can't find the container with id 62c9679032e1ef110df0bb3ef17bd72b4e93a4b94bb23044b6d1cf7707059018 Nov 02 21:22:54.753381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:54.753283 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" event={"ID":"94522fad-8716-4425-85ed-dba0714e23ad","Type":"ContainerStarted","Data":"7d635c3a0ca95aa371e0083ea6f37a0ad2289dadaedb16e6fb41e17d88a90947"} Nov 02 21:22:54.753515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:54.753382 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" event={"ID":"94522fad-8716-4425-85ed-dba0714e23ad","Type":"ContainerStarted","Data":"62c9679032e1ef110df0bb3ef17bd72b4e93a4b94bb23044b6d1cf7707059018"} Nov 02 21:22:54.860766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:54.860721 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" podStartSLOduration=1.263143614 podStartE2EDuration="1.860707578s" totalImagesPullingTime="597.563964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:22:54.858382847 +0000 UTC m=+600.834680603" watchObservedRunningTime="2026-11-02 21:22:54.860707578 +0000 UTC m=+600.837005353" Nov 02 21:22:55.529927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:55.529890 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:22:55.918066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:55.918034 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:55.930477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:55.930457 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:56.774001 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:56.773967 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd" Nov 02 21:22:59.513574 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:59.513539 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:59.525984 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:59.525959 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:22:59.787407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:22:59.787333 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tpgnk" Nov 02 21:23:03.529588 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:03.529555 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:23:03.541974 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:03.541946 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:23:03.803886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:03.803810 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-svskm" Nov 02 21:23:07.043300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.043268 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5"] Nov 02 21:23:07.046875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.046850 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.131720 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.131687 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 21:23:07.142946 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.142916 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5"] Nov 02 21:23:07.229696 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.229660 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.229871 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.229717 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.229871 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.229837 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.229942 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.229903 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.229979 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.229971 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.230018 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.229991 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqsf\" (UniqueName: \"kubernetes.io/projected/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-kube-api-access-pkqsf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.330552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.330455 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.330552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.330512 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.330771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.330559 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqsf\" (UniqueName: \"kubernetes.io/projected/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-kube-api-access-pkqsf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.330771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.330609 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.330771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.330641 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.330771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.330657 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.331142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.331109 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.331142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.331131 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.331308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.331171 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.332998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.332968 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.333124 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.333110 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.382819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.382797 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqsf\" (UniqueName: \"kubernetes.io/projected/28ac4cf9-d63f-40ee-b6b6-fad22e39c631-kube-api-access-pkqsf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-wgtj5\" (UID: \"28ac4cf9-d63f-40ee-b6b6-fad22e39c631\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.659417 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.659385 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:07.845046 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.845016 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282"] Nov 02 21:23:07.849840 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.849820 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:07.898337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.898312 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 21:23:07.898445 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.898369 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-445mt\"" Nov 02 21:23:07.946770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:07.946706 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282"] Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038564 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038626 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038662 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038717 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc6z9\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-kube-api-access-jc6z9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038824 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038856 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038899 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.038981 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.043643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.039008 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.062605 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.062576 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5"] Nov 02 21:23:08.070108 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:23:08.070083 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ac4cf9_d63f_40ee_b6b6_fad22e39c631.slice/crio-95e3bcdd13f7f31bd79c9130e8ce065a63f12bccf31d13aa9ac0312e2387a659 WatchSource:0}: Error finding container 95e3bcdd13f7f31bd79c9130e8ce065a63f12bccf31d13aa9ac0312e2387a659: Status 404 returned error can't find the container with id 95e3bcdd13f7f31bd79c9130e8ce065a63f12bccf31d13aa9ac0312e2387a659 Nov 02 21:23:08.139893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.139861 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.139893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.139894 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.139925 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.139963 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.139980 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.139999 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140015 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140033 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140080 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc6z9\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-kube-api-access-jc6z9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140305 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140346 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140437 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140639 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140616 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.140750 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.140734 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.142364 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.142344 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.142477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.142461 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.192365 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.192342 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.192488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.192471 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc6z9\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-kube-api-access-jc6z9\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7x282\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.463446 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.463412 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:08.818847 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.818811 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" event={"ID":"28ac4cf9-d63f-40ee-b6b6-fad22e39c631","Type":"ContainerStarted","Data":"b308a894272e75835215321d50b10384216264a6264d4b65c464a351ea6e7392"} Nov 02 21:23:08.818986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.818866 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" event={"ID":"28ac4cf9-d63f-40ee-b6b6-fad22e39c631","Type":"ContainerStarted","Data":"95e3bcdd13f7f31bd79c9130e8ce065a63f12bccf31d13aa9ac0312e2387a659"} Nov 02 21:23:08.924670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.924619 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" podStartSLOduration=1.290101046 podStartE2EDuration="1.924605404s" totalImagesPullingTime="634.504358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:08.92374349 +0000 UTC m=+614.900041267" watchObservedRunningTime="2026-11-02 21:23:08.924605404 +0000 UTC m=+614.900903178" Nov 02 21:23:08.936258 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:08.936232 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282"] Nov 02 21:23:08.943987 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:23:08.943964 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312e36a7_e1b0_4eb0_9f4e_9ecd047a6e8c.slice/crio-b1451650f73755833a7bb065d2abffad2a38924fb163910ef53b0c3fbd7192de WatchSource:0}: Error finding container b1451650f73755833a7bb065d2abffad2a38924fb163910ef53b0c3fbd7192de: Status 404 returned error can't find the container with id b1451650f73755833a7bb065d2abffad2a38924fb163910ef53b0c3fbd7192de Nov 02 21:23:09.660151 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:09.660116 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:09.823888 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:09.823854 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" event={"ID":"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c","Type":"ContainerStarted","Data":"b1451650f73755833a7bb065d2abffad2a38924fb163910ef53b0c3fbd7192de"} Nov 02 21:23:10.633290 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:10.633252 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:23:13.842470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:13.842431 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:23:13.843036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:13.842508 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:23:13.843036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:13.842562 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:23:14.854011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:14.853969 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" event={"ID":"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c","Type":"ContainerStarted","Data":"d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded"} Nov 02 21:23:15.054711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:15.054648 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podStartSLOduration=3.159121425 podStartE2EDuration="8.054634299s" totalImagesPullingTime="4.895512874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:15.007668437 +0000 UTC m=+620.983966223" watchObservedRunningTime="2026-11-02 21:23:15.054634299 +0000 UTC m=+621.030932109" Nov 02 21:23:15.464488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:15.464450 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:15.465939 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:15.465915 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7x282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:23:15.466057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:15.465964 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:23:15.553077 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:23:15.553042 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:23:16.464473 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:16.464438 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7x282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:23:16.464890 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:16.464487 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:23:17.463962 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:17.463921 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7x282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:23:17.464163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:17.463971 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:23:17.660336 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:17.660296 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:17.672418 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:17.672393 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:17.880951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:17.880924 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-wgtj5" Nov 02 21:23:18.464233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:18.464198 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7x282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:23:18.464407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:18.464250 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:23:19.464153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:19.464115 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7x282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:23:19.464586 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:19.464175 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:23:20.464214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:20.464174 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7x282 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 21:23:20.464616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:20.464240 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 21:23:22.031838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:22.031802 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:22.032287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:22.032203 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:22.033017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:22.032997 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:23:27.118271 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.118235 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv"] Nov 02 21:23:27.124765 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.124745 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.173582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.173552 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 21:23:27.174958 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.174927 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv"] Nov 02 21:23:27.218727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.218703 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-2vc2g\"" Nov 02 21:23:27.218727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.218722 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 21:23:27.245033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.245008 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76618ea-0665-47ee-b896-731f8923967f-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-8s6nv\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.245122 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.245042 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssz26\" (UniqueName: \"kubernetes.io/projected/e76618ea-0665-47ee-b896-731f8923967f-kube-api-access-ssz26\") pod \"ai-gateway-controller-7cb8f6bb64-8s6nv\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.346660 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.346632 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76618ea-0665-47ee-b896-731f8923967f-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-8s6nv\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.346777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.346670 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssz26\" (UniqueName: \"kubernetes.io/projected/e76618ea-0665-47ee-b896-731f8923967f-kube-api-access-ssz26\") pod \"ai-gateway-controller-7cb8f6bb64-8s6nv\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.349121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.349100 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76618ea-0665-47ee-b896-731f8923967f-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-8s6nv\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.400087 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.400030 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssz26\" (UniqueName: \"kubernetes.io/projected/e76618ea-0665-47ee-b896-731f8923967f-kube-api-access-ssz26\") pod \"ai-gateway-controller-7cb8f6bb64-8s6nv\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.438857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.438834 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:27.804794 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.804722 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv"] Nov 02 21:23:27.812274 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:23:27.812244 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76618ea_0665_47ee_b896_731f8923967f.slice/crio-ebe4cf6b1e27bc99f431f5e7cc0fa7aaf6c2453db4b3ff4534c96897155b02d0 WatchSource:0}: Error finding container ebe4cf6b1e27bc99f431f5e7cc0fa7aaf6c2453db4b3ff4534c96897155b02d0: Status 404 returned error can't find the container with id ebe4cf6b1e27bc99f431f5e7cc0fa7aaf6c2453db4b3ff4534c96897155b02d0 Nov 02 21:23:27.813738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.813720 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:23:27.918788 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:27.918755 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" event={"ID":"e76618ea-0665-47ee-b896-731f8923967f","Type":"ContainerStarted","Data":"ebe4cf6b1e27bc99f431f5e7cc0fa7aaf6c2453db4b3ff4534c96897155b02d0"} Nov 02 21:23:30.935289 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:30.935254 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" event={"ID":"e76618ea-0665-47ee-b896-731f8923967f","Type":"ContainerStarted","Data":"1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557"} Nov 02 21:23:30.935708 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:30.935404 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:31.039799 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:31.039742 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" podStartSLOduration=1.5786585610000001 podStartE2EDuration="4.039729493s" totalImagesPullingTime="2.461070932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:31.039309495 +0000 UTC m=+637.015607269" watchObservedRunningTime="2026-11-02 21:23:31.039729493 +0000 UTC m=+637.016027268" Nov 02 21:23:34.585309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.585272 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f"] Nov 02 21:23:34.589377 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.589354 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.640837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.640804 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 21:23:34.640990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.640814 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jxdx2\"" Nov 02 21:23:34.642684 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.642656 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f"] Nov 02 21:23:34.716304 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.716271 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgv2\" (UniqueName: \"kubernetes.io/projected/9ea1fa27-9f65-44e7-b017-c0837b39a362-kube-api-access-ktgv2\") pod \"payload-pre-processing-7b98bb678c-lcj5f\" (UID: \"9ea1fa27-9f65-44e7-b017-c0837b39a362\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.716479 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.716399 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea1fa27-9f65-44e7-b017-c0837b39a362-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lcj5f\" (UID: \"9ea1fa27-9f65-44e7-b017-c0837b39a362\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.730177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.730146 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jmk5x"] Nov 02 21:23:34.733868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.733846 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:34.788140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.788106 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jmk5x"] Nov 02 21:23:34.818180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.818143 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgv2\" (UniqueName: \"kubernetes.io/projected/9ea1fa27-9f65-44e7-b017-c0837b39a362-kube-api-access-ktgv2\") pod \"payload-pre-processing-7b98bb678c-lcj5f\" (UID: \"9ea1fa27-9f65-44e7-b017-c0837b39a362\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.818354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.818214 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjbb\" (UniqueName: \"kubernetes.io/projected/eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6-kube-api-access-8rjbb\") pod \"payload-processing-55fbbbf9dc-jmk5x\" (UID: \"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:34.818354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.818272 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea1fa27-9f65-44e7-b017-c0837b39a362-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lcj5f\" (UID: \"9ea1fa27-9f65-44e7-b017-c0837b39a362\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.818354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.818298 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-jmk5x\" (UID: \"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:34.818995 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.818976 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea1fa27-9f65-44e7-b017-c0837b39a362-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-lcj5f\" (UID: \"9ea1fa27-9f65-44e7-b017-c0837b39a362\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.871075 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.871038 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgv2\" (UniqueName: \"kubernetes.io/projected/9ea1fa27-9f65-44e7-b017-c0837b39a362-kube-api-access-ktgv2\") pod \"payload-pre-processing-7b98bb678c-lcj5f\" (UID: \"9ea1fa27-9f65-44e7-b017-c0837b39a362\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.901809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.901786 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:34.919266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.919238 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjbb\" (UniqueName: \"kubernetes.io/projected/eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6-kube-api-access-8rjbb\") pod \"payload-processing-55fbbbf9dc-jmk5x\" (UID: \"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:34.919392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.919320 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-jmk5x\" (UID: \"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:34.920342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.920319 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-jmk5x\" (UID: \"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:34.975580 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:34.975552 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjbb\" (UniqueName: \"kubernetes.io/projected/eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6-kube-api-access-8rjbb\") pod \"payload-processing-55fbbbf9dc-jmk5x\" (UID: \"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:35.048205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:35.048170 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:35.273552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:35.273440 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f"] Nov 02 21:23:35.281399 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:23:35.281372 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea1fa27_9f65_44e7_b017_c0837b39a362.slice/crio-fce5c41df9680a7b774a018d77e92126ea0dcf036419f672e43a639ddd4efa66 WatchSource:0}: Error finding container fce5c41df9680a7b774a018d77e92126ea0dcf036419f672e43a639ddd4efa66: Status 404 returned error can't find the container with id fce5c41df9680a7b774a018d77e92126ea0dcf036419f672e43a639ddd4efa66 Nov 02 21:23:35.416446 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:35.416412 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jmk5x"] Nov 02 21:23:35.424839 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:23:35.424815 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5bf3d8_3b15_4597_9f54_83bd1faa6ac6.slice/crio-867f9ea1a64684d0d5520099d14d5c0e9a8be660cceca25cb6e5c2f1587c9c4c WatchSource:0}: Error finding container 867f9ea1a64684d0d5520099d14d5c0e9a8be660cceca25cb6e5c2f1587c9c4c: Status 404 returned error can't find the container with id 867f9ea1a64684d0d5520099d14d5c0e9a8be660cceca25cb6e5c2f1587c9c4c Nov 02 21:23:35.965150 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:35.965105 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" event={"ID":"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6","Type":"ContainerStarted","Data":"867f9ea1a64684d0d5520099d14d5c0e9a8be660cceca25cb6e5c2f1587c9c4c"} Nov 02 21:23:35.966676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:35.966645 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" event={"ID":"9ea1fa27-9f65-44e7-b017-c0837b39a362","Type":"ContainerStarted","Data":"fce5c41df9680a7b774a018d77e92126ea0dcf036419f672e43a639ddd4efa66"} Nov 02 21:23:38.984935 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:38.984899 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" event={"ID":"9ea1fa27-9f65-44e7-b017-c0837b39a362","Type":"ContainerStarted","Data":"7b8a25bfb0db22267d047f4c6a54c579f13d00dc5e4f20ea87c05f32cd6938be"} Nov 02 21:23:38.985368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:38.985040 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:38.986484 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:38.986461 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" event={"ID":"eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6","Type":"ContainerStarted","Data":"83c7e856ea491e7be348bc94b6071ceae531d61ab6374b585bd89026229a1f2a"} Nov 02 21:23:38.986603 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:38.986559 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:39.089419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:39.089372 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" podStartSLOduration=2.272509741 podStartE2EDuration="5.089356256s" totalImagesPullingTime="2.816846515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:39.088344458 +0000 UTC m=+645.064642233" watchObservedRunningTime="2026-11-02 21:23:39.089356256 +0000 UTC m=+645.065654023" Nov 02 21:23:39.196055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:39.196004 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" podStartSLOduration=2.179767742 podStartE2EDuration="5.195991742s" totalImagesPullingTime="3.016224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:39.193341934 +0000 UTC m=+645.169639709" watchObservedRunningTime="2026-11-02 21:23:39.195991742 +0000 UTC m=+645.172289516" Nov 02 21:23:41.943589 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:41.943557 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:45.531309 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:23:45.531272 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:23:49.993812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:49.993780 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-lcj5f" Nov 02 21:23:49.994244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:49.993850 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-jmk5x" Nov 02 21:23:53.190410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.190380 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9"] Nov 02 21:23:53.200126 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.200102 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.295789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.295753 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.295943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.295797 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.295943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.295894 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.295943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.295930 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.296083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.295996 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.296083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.296030 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.296083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.296071 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.296206 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.296175 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79616d36-9d8b-4e32-bcbf-01ec99576118-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.296253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.296241 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkfb\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-kube-api-access-qbkfb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.296615 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.296597 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9"] Nov 02 21:23:53.396837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396807 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.396837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396838 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396858 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396875 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396902 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396924 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396955 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.396988 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79616d36-9d8b-4e32-bcbf-01ec99576118-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.397062 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkfb\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-kube-api-access-qbkfb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397448 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.397339 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397448 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.397372 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.397472 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.397594 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.397834 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.397815 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79616d36-9d8b-4e32-bcbf-01ec99576118-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.399254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.399236 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.399392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.399376 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.450050 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.449988 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.450154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.450052 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkfb\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-kube-api-access-qbkfb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-j9dd9\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.512768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.512744 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:53.934871 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.934839 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9"] Nov 02 21:23:53.942950 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:23:53.942923 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79616d36_9d8b_4e32_bcbf_01ec99576118.slice/crio-35c174c3a3ff99ffe5bfc910b40a9acf68f379eee77e8bf8e42b497db11d8075 WatchSource:0}: Error finding container 35c174c3a3ff99ffe5bfc910b40a9acf68f379eee77e8bf8e42b497db11d8075: Status 404 returned error can't find the container with id 35c174c3a3ff99ffe5bfc910b40a9acf68f379eee77e8bf8e42b497db11d8075 Nov 02 21:23:53.945119 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.945086 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:23:53.945210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.945148 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:23:53.945210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:53.945177 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:23:54.060886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:54.060851 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" event={"ID":"79616d36-9d8b-4e32-bcbf-01ec99576118","Type":"ContainerStarted","Data":"694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6"} Nov 02 21:23:54.061033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:54.060927 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" event={"ID":"79616d36-9d8b-4e32-bcbf-01ec99576118","Type":"ContainerStarted","Data":"35c174c3a3ff99ffe5bfc910b40a9acf68f379eee77e8bf8e42b497db11d8075"} Nov 02 21:23:54.169567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:54.169500 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podStartSLOduration=1.16948731 podStartE2EDuration="1.16948731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:23:54.167664178 +0000 UTC m=+660.143961955" watchObservedRunningTime="2026-11-02 21:23:54.16948731 +0000 UTC m=+660.145785084" Nov 02 21:23:54.513683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:54.513648 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:23:55.514475 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:55.514439 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:23:55.514932 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:55.514493 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 02 21:23:56.514132 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:56.514095 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:23:56.514318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:56.514152 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 02 21:23:57.250115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.249988 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv"] Nov 02 21:23:57.250656 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.250230 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" podUID="e76618ea-0665-47ee-b896-731f8923967f" containerName="manager" containerID="cri-o://1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557" gracePeriod=30 Nov 02 21:23:57.507669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.507609 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:57.513439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.513416 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:23:57.513542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.513459 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 02 21:23:57.560262 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.560232 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76618ea-0665-47ee-b896-731f8923967f-kube-api-access-ssz26\" (UniqueName: \"kubernetes.io/projected/e76618ea-0665-47ee-b896-731f8923967f-kube-api-access-ssz26\") pod \"e76618ea-0665-47ee-b896-731f8923967f\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " Nov 02 21:23:57.560430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.560322 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76618ea-0665-47ee-b896-731f8923967f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76618ea-0665-47ee-b896-731f8923967f-metrics-certs\") pod \"e76618ea-0665-47ee-b896-731f8923967f\" (UID: \"e76618ea-0665-47ee-b896-731f8923967f\") " Nov 02 21:23:57.562490 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.562460 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76618ea-0665-47ee-b896-731f8923967f-kube-api-access-ssz26" pod "e76618ea-0665-47ee-b896-731f8923967f" (UID: "e76618ea-0665-47ee-b896-731f8923967f"). InnerVolumeSpecName "kube-api-access-ssz26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:23:57.562490 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.562467 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76618ea-0665-47ee-b896-731f8923967f-metrics-certs" pod "e76618ea-0665-47ee-b896-731f8923967f" (UID: "e76618ea-0665-47ee-b896-731f8923967f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:23:57.661704 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.661668 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssz26\" (UniqueName: \"kubernetes.io/projected/e76618ea-0665-47ee-b896-731f8923967f-kube-api-access-ssz26\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:57.661704 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:57.661696 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e76618ea-0665-47ee-b896-731f8923967f-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:23:58.081329 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.081294 4510 generic.go:356] "Generic (PLEG): container finished" podID="e76618ea-0665-47ee-b896-731f8923967f" containerID="1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557" exitCode=0 Nov 02 21:23:58.081530 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.081338 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" Nov 02 21:23:58.081530 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.081383 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" event={"ID":"e76618ea-0665-47ee-b896-731f8923967f","Type":"ContainerDied","Data":"1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557"} Nov 02 21:23:58.081530 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.081434 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv" event={"ID":"e76618ea-0665-47ee-b896-731f8923967f","Type":"ContainerDied","Data":"ebe4cf6b1e27bc99f431f5e7cc0fa7aaf6c2453db4b3ff4534c96897155b02d0"} Nov 02 21:23:58.081530 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.081456 4510 scope.go:122] "RemoveContainer" containerID="1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557" Nov 02 21:23:58.090465 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.090437 4510 scope.go:122] "RemoveContainer" containerID="1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557" Nov 02 21:23:58.090738 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:23:58.090720 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557\": container with ID starting with 1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557 not found: ID does not exist" containerID="1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557" Nov 02 21:23:58.090792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.090748 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557"} err="failed to get container status \"1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557\": rpc error: code = NotFound desc = could not find container \"1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557\": container with ID starting with 1af8cd2fd352532d537899e8e38f78879cfbbf5654160352867902a352908557 not found: ID does not exist" Nov 02 21:23:58.249137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.249096 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv"] Nov 02 21:23:58.251781 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.251756 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-8s6nv"] Nov 02 21:23:58.513576 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.513515 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:23:58.513757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:58.513596 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 02 21:23:59.514018 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:59.513982 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:23:59.514455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:23:59.514038 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 02 21:24:00.152138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:00.152105 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76618ea-0665-47ee-b896-731f8923967f" path="/var/lib/kubelet/pods/e76618ea-0665-47ee-b896-731f8923967f/volumes" Nov 02 21:24:00.513545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:00.513446 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 02 21:24:00.513545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:00.513492 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 02 21:24:02.171882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:02.171851 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:24:02.172378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:02.172301 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:24:02.173122 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:02.173098 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:24:15.529433 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:24:15.529355 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:24:27.873775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:27.873737 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5"] Nov 02 21:24:27.874356 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:27.874224 4510 cpu_manager.go:414] "containerMap: removing container" podUID="e76618ea-0665-47ee-b896-731f8923967f" containerName="manager" Nov 02 21:24:27.874356 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:27.874237 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76618ea-0665-47ee-b896-731f8923967f" containerName="manager" Nov 02 21:24:27.874356 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:27.874344 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76618ea-0665-47ee-b896-731f8923967f" containerName="manager" Nov 02 21:24:27.880931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:27.880907 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:27.935831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:27.935797 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5"] Nov 02 21:24:27.978343 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:27.978316 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9a6cf2-openshift-default-dockercfg-d2zhq\"" Nov 02 21:24:28.045379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045341 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045584 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045394 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj6fm\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-kube-api-access-nj6fm\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045584 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045500 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045584 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045562 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045645 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045676 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045704 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045803 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045722 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.045864 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.045848 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b023881e-83cb-44b7-9db4-246dd5665b6a-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.146707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146637 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.146707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146682 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b023881e-83cb-44b7-9db4-246dd5665b6a-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.146927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146735 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.146927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146755 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj6fm\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-kube-api-access-nj6fm\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.146927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146886 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146931 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146966 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.146993 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.147034 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147260 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.147140 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147260 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.147188 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.147359 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147420 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.147388 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.147591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.147568 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b023881e-83cb-44b7-9db4-246dd5665b6a-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.149380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.149360 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.149480 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.149445 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.199876 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.199846 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.199876 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.199865 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj6fm\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-kube-api-access-nj6fm\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.494247 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.494167 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:28.754085 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.754015 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx"] Nov 02 21:24:28.759175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.759154 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.815034 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.814999 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx"] Nov 02 21:24:28.855918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.855889 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6v2h\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-kube-api-access-b6v2h\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.855930 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.856006 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.856028 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.856043 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.856074 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.856090 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.856231 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.856330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.856278 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.895923 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.895883 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4e2b27-openshift-default-dockercfg-knh48\"" Nov 02 21:24:28.923366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.923338 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5"] Nov 02 21:24:28.930094 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:28.930072 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb023881e_83cb_44b7_9db4_246dd5665b6a.slice/crio-2479b626410d030d952dcd69c4cfaa03d866d08b16d841def3d2c22ce4209132 WatchSource:0}: Error finding container 2479b626410d030d952dcd69c4cfaa03d866d08b16d841def3d2c22ce4209132: Status 404 returned error can't find the container with id 2479b626410d030d952dcd69c4cfaa03d866d08b16d841def3d2c22ce4209132 Nov 02 21:24:28.932362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.932333 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:28.932450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.932398 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:28.932450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.932429 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:28.956846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.956820 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6v2h\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-kube-api-access-b6v2h\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.956960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.956870 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.956960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.956931 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.956986 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957015 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957054 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957072 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957096 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957154 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957397 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957454 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957490 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957578 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.957802 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.957783 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.959221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.959199 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:28.959300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:28.959274 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:29.009294 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.009235 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:29.009379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.009326 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6v2h\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-kube-api-access-b6v2h\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:29.046304 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.046270 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6"] Nov 02 21:24:29.050228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.050209 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.073083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.073058 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:29.101483 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.101445 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-284b8e-openshift-default-dockercfg-w2r92\"" Nov 02 21:24:29.110878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.110849 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6"] Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161417 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161458 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161495 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161577 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161615 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161649 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161744 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161807 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.162277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.161846 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmqzs\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-kube-api-access-jmqzs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.243869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.243836 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" event={"ID":"b023881e-83cb-44b7-9db4-246dd5665b6a","Type":"ContainerStarted","Data":"2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3"} Nov 02 21:24:29.244083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.243889 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" event={"ID":"b023881e-83cb-44b7-9db4-246dd5665b6a","Type":"ContainerStarted","Data":"2479b626410d030d952dcd69c4cfaa03d866d08b16d841def3d2c22ce4209132"} Nov 02 21:24:29.263386 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263306 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263386 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263354 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263603 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263431 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263603 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263483 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263603 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263535 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmqzs\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-kube-api-access-jmqzs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263602 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263625 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263660 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263707 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263771 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.263953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.263784 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.264081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.264062 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.264471 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.264439 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.264629 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.264515 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.266304 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.266279 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.266392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.266363 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.316607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.316515 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.316734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.316668 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmqzs\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-kube-api-access-jmqzs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.355715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.355670 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podStartSLOduration=2.35565488 podStartE2EDuration="2.35565488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:29.353402882 +0000 UTC m=+695.329700680" watchObservedRunningTime="2026-11-02 21:24:29.35565488 +0000 UTC m=+695.331952655" Nov 02 21:24:29.364612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.364587 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:29.472636 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.472600 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx"] Nov 02 21:24:29.479397 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:29.479372 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b68b66b_eb3b_4eff_8e07_c36076d211ee.slice/crio-8c31f089fb1e42d424367bc0d90cb716bf7304f720f20f50dee6b72aa1078331 WatchSource:0}: Error finding container 8c31f089fb1e42d424367bc0d90cb716bf7304f720f20f50dee6b72aa1078331: Status 404 returned error can't find the container with id 8c31f089fb1e42d424367bc0d90cb716bf7304f720f20f50dee6b72aa1078331 Nov 02 21:24:29.482411 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.482376 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:29.482543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.482458 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:29.482543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.482500 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:29.494934 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.494911 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:29.753744 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.753706 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6"] Nov 02 21:24:29.766095 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:29.766061 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9f15c8_aea4_4c05_9945_301b760e5fb7.slice/crio-a6ff963845e3b12e7b78998748bb3d6e5eaeae9196381d0c75b96204775bfda8 WatchSource:0}: Error finding container a6ff963845e3b12e7b78998748bb3d6e5eaeae9196381d0c75b96204775bfda8: Status 404 returned error can't find the container with id a6ff963845e3b12e7b78998748bb3d6e5eaeae9196381d0c75b96204775bfda8 Nov 02 21:24:29.769048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.769010 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:29.769166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.769098 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:29.769166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:29.769144 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:24:30.250960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.250919 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" event={"ID":"0a9f15c8-aea4-4c05-9945-301b760e5fb7","Type":"ContainerStarted","Data":"d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7"} Nov 02 21:24:30.251432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.250974 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" event={"ID":"0a9f15c8-aea4-4c05-9945-301b760e5fb7","Type":"ContainerStarted","Data":"a6ff963845e3b12e7b78998748bb3d6e5eaeae9196381d0c75b96204775bfda8"} Nov 02 21:24:30.252607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.252582 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" event={"ID":"7b68b66b-eb3b-4eff-8e07-c36076d211ee","Type":"ContainerStarted","Data":"7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42"} Nov 02 21:24:30.252681 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.252623 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" event={"ID":"7b68b66b-eb3b-4eff-8e07-c36076d211ee","Type":"ContainerStarted","Data":"8c31f089fb1e42d424367bc0d90cb716bf7304f720f20f50dee6b72aa1078331"} Nov 02 21:24:30.361973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.361915 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podStartSLOduration=1.361902864 podStartE2EDuration="1.361902864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:30.359494786 +0000 UTC m=+696.335792563" watchObservedRunningTime="2026-11-02 21:24:30.361902864 +0000 UTC m=+696.338200639" Nov 02 21:24:30.365204 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.365174 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:30.370416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.370392 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:30.467300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.467231 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podStartSLOduration=2.467213333 podStartE2EDuration="2.467213333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:30.465598255 +0000 UTC m=+696.441896030" watchObservedRunningTime="2026-11-02 21:24:30.467213333 +0000 UTC m=+696.443511109" Nov 02 21:24:30.500138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:30.500107 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:31.073492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.073456 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:31.078944 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.078915 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:31.263899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.263862 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:31.264380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.264250 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:31.264380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.264293 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:31.265064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.265025 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:24:31.265186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.265087 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:24:31.265350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:31.265332 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:24:38.255246 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.255213 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467"] Nov 02 21:24:38.259259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.259240 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.309858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.309832 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9a6cf2\"" Nov 02 21:24:38.313887 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.313859 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467"] Nov 02 21:24:38.356353 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.356324 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9a6cf2-dockercfg-x6qdp\"" Nov 02 21:24:38.358771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.358749 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.358843 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.358809 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsj7\" (UniqueName: \"kubernetes.io/projected/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-kube-api-access-vnsj7\") pod \"payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.459794 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.459756 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.459955 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.459860 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsj7\" (UniqueName: \"kubernetes.io/projected/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-kube-api-access-vnsj7\") pod \"payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.460487 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.460461 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.512667 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.512603 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsj7\" (UniqueName: \"kubernetes.io/projected/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-kube-api-access-vnsj7\") pod \"payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.571806 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.571774 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:38.940093 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:38.940066 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467"] Nov 02 21:24:38.948079 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:38.948051 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b30ae4c_6c6e_4975_9c84_6d76ba5c5544.slice/crio-4f027ae5d8bef6dc375f1c1583106515c50e3849139a36db35a0df5799273597 WatchSource:0}: Error finding container 4f027ae5d8bef6dc375f1c1583106515c50e3849139a36db35a0df5799273597: Status 404 returned error can't find the container with id 4f027ae5d8bef6dc375f1c1583106515c50e3849139a36db35a0df5799273597 Nov 02 21:24:39.308774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:39.308689 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" event={"ID":"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544","Type":"ContainerStarted","Data":"0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b"} Nov 02 21:24:39.308774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:39.308741 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" event={"ID":"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544","Type":"ContainerStarted","Data":"4f027ae5d8bef6dc375f1c1583106515c50e3849139a36db35a0df5799273597"} Nov 02 21:24:39.309260 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:39.308810 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:39.414252 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:39.414202 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" podStartSLOduration=1.414189313 podStartE2EDuration="1.414189313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:39.412505156 +0000 UTC m=+705.388802932" watchObservedRunningTime="2026-11-02 21:24:39.414189313 +0000 UTC m=+705.390487088" Nov 02 21:24:42.662263 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.662228 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j"] Nov 02 21:24:42.666244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.666223 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:42.701337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.701314 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wnn\" (UniqueName: \"kubernetes.io/projected/d44b2c19-9cac-4549-bbbb-414729843150-kube-api-access-s5wnn\") pod \"payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:42.701482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.701459 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b2c19-9cac-4549-bbbb-414729843150-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:42.763035 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.763004 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ef6b43-dockercfg-mpvnb\"" Nov 02 21:24:42.763195 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.763060 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ef6b43\"" Nov 02 21:24:42.765354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.765330 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j"] Nov 02 21:24:42.802301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.802266 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b2c19-9cac-4549-bbbb-414729843150-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:42.802423 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.802317 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wnn\" (UniqueName: \"kubernetes.io/projected/d44b2c19-9cac-4549-bbbb-414729843150-kube-api-access-s5wnn\") pod \"payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:42.803008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.802987 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b2c19-9cac-4549-bbbb-414729843150-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:42.857428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.857404 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wnn\" (UniqueName: \"kubernetes.io/projected/d44b2c19-9cac-4549-bbbb-414729843150-kube-api-access-s5wnn\") pod \"payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:42.979059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:42.978988 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:43.254463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.254394 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7"] Nov 02 21:24:43.259296 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.259275 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.308471 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.308439 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-metrics-certs\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.308623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.308514 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.308623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.308587 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.308742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.308722 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrcq\" (UniqueName: \"kubernetes.io/projected/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-kube-api-access-rkrcq\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.309487 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.309467 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 21:24:43.309581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.309476 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-4d9ea0-serving-cert\"" Nov 02 21:24:43.355077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.355052 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47p47\"" Nov 02 21:24:43.357352 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.357327 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7"] Nov 02 21:24:43.370551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.370513 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j"] Nov 02 21:24:43.378019 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:43.377995 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44b2c19_9cac_4549_bbbb_414729843150.slice/crio-1eae32f39bec983e2a27964d58806a0b7ff794aa3eb4d0196031ae7b2353d837 WatchSource:0}: Error finding container 1eae32f39bec983e2a27964d58806a0b7ff794aa3eb4d0196031ae7b2353d837: Status 404 returned error can't find the container with id 1eae32f39bec983e2a27964d58806a0b7ff794aa3eb4d0196031ae7b2353d837 Nov 02 21:24:43.417792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.410055 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrcq\" (UniqueName: \"kubernetes.io/projected/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-kube-api-access-rkrcq\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.417792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.410176 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-metrics-certs\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.417792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.410267 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.417792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.410325 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.417792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.411201 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.417792 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:24:43.411922 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-serving-cert: secret "maas-api-e2e-worker-w2-4d9ea0-serving-cert" not found Nov 02 21:24:43.417792 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:24:43.412001 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls podName:2c674c14-2c24-4aa7-88f7-f2d9d0364bf0 nodeName:}" failed. No retries permitted until 2026-11-02 21:24:43.911981234 +0000 UTC m=+709.888279000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls") pod "maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" (UID: "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0") : secret "maas-api-e2e-worker-w2-4d9ea0-serving-cert" not found Nov 02 21:24:43.418830 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.418808 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-metrics-certs\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.463213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.463192 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrcq\" (UniqueName: \"kubernetes.io/projected/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-kube-api-access-rkrcq\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.916266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.916234 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:43.918374 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:43.918350 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:44.172407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:44.172322 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:44.335122 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:44.335084 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" event={"ID":"d44b2c19-9cac-4549-bbbb-414729843150","Type":"ContainerStarted","Data":"df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082"} Nov 02 21:24:44.335301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:44.335140 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" event={"ID":"d44b2c19-9cac-4549-bbbb-414729843150","Type":"ContainerStarted","Data":"1eae32f39bec983e2a27964d58806a0b7ff794aa3eb4d0196031ae7b2353d837"} Nov 02 21:24:44.335301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:44.335167 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:44.440821 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:44.440739 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" podStartSLOduration=2.440726398 podStartE2EDuration="2.440726398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:44.440708518 +0000 UTC m=+710.417006294" watchObservedRunningTime="2026-11-02 21:24:44.440726398 +0000 UTC m=+710.417024173" Nov 02 21:24:44.599481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:44.599450 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7"] Nov 02 21:24:44.606990 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:44.606965 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c674c14_2c24_4aa7_88f7_f2d9d0364bf0.slice/crio-fea10c5adbe2195ce1efeea1d3991d025135d6c21c40c1f59d1abdc5fe1d7416 WatchSource:0}: Error finding container fea10c5adbe2195ce1efeea1d3991d025135d6c21c40c1f59d1abdc5fe1d7416: Status 404 returned error can't find the container with id fea10c5adbe2195ce1efeea1d3991d025135d6c21c40c1f59d1abdc5fe1d7416 Nov 02 21:24:45.341015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:45.340971 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" event={"ID":"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0","Type":"ContainerStarted","Data":"fea10c5adbe2195ce1efeea1d3991d025135d6c21c40c1f59d1abdc5fe1d7416"} Nov 02 21:24:45.535125 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:24:45.535093 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:24:46.347866 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:46.347824 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" event={"ID":"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0","Type":"ContainerStarted","Data":"f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a"} Nov 02 21:24:46.348293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:46.347954 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:46.456328 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:46.456278 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" podStartSLOduration=2.533774482 podStartE2EDuration="3.45626408s" totalImagesPullingTime="922.489598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:46.453749691 +0000 UTC m=+712.430047466" watchObservedRunningTime="2026-11-02 21:24:46.45626408 +0000 UTC m=+712.432561896" Nov 02 21:24:46.711596 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:46.711563 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:24:47.593634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.593601 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv"] Nov 02 21:24:47.598500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.598480 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:47.651663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.651621 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv"] Nov 02 21:24:47.657509 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.657481 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:47.657759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.657738 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgx62\" (UniqueName: \"kubernetes.io/projected/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-kube-api-access-hgx62\") pod \"payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:47.692964 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.692926 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c2dd3b\"" Nov 02 21:24:47.693113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.692982 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c2dd3b-dockercfg-brg4g\"" Nov 02 21:24:47.759553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.759499 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:47.759699 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.759592 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgx62\" (UniqueName: \"kubernetes.io/projected/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-kube-api-access-hgx62\") pod \"payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:47.760214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.760196 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:47.811986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.811962 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgx62\" (UniqueName: \"kubernetes.io/projected/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-kube-api-access-hgx62\") pod \"payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:47.911442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:47.911417 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:48.269618 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.269546 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw"] Nov 02 21:24:48.274448 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.274428 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.278110 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.278091 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv"] Nov 02 21:24:48.286538 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:48.286499 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebf0ea5_1a63_4c90_8870_d835e4ff1142.slice/crio-258ba8d3a40ca27f5a0e2c09dde28e97dd3c88e2980e7cbd95a6c3e0b317438f WatchSource:0}: Error finding container 258ba8d3a40ca27f5a0e2c09dde28e97dd3c88e2980e7cbd95a6c3e0b317438f: Status 404 returned error can't find the container with id 258ba8d3a40ca27f5a0e2c09dde28e97dd3c88e2980e7cbd95a6c3e0b317438f Nov 02 21:24:48.325374 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.325350 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c2dd3b-serving-cert\"" Nov 02 21:24:48.328437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.328377 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw"] Nov 02 21:24:48.366678 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.366657 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.366771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.366707 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldpwv\" (UniqueName: \"kubernetes.io/projected/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-kube-api-access-ldpwv\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.366888 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.366860 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-metrics-certs\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.366985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.366967 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.368066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.368044 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" event={"ID":"4ebf0ea5-1a63-4c90-8870-d835e4ff1142","Type":"ContainerStarted","Data":"9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b"} Nov 02 21:24:48.368125 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.368094 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" event={"ID":"4ebf0ea5-1a63-4c90-8870-d835e4ff1142","Type":"ContainerStarted","Data":"258ba8d3a40ca27f5a0e2c09dde28e97dd3c88e2980e7cbd95a6c3e0b317438f"} Nov 02 21:24:48.368125 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.368118 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:24:48.468140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.468108 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-metrics-certs\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.468297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.468265 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.468379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.468355 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.468444 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.468428 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldpwv\" (UniqueName: \"kubernetes.io/projected/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-kube-api-access-ldpwv\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.468556 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:24:48.468504 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-serving-cert: secret "maas-api-e2e-worker-w1-c2dd3b-serving-cert" not found Nov 02 21:24:48.468631 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:24:48.468588 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls podName:110c5f36-3efa-4fdb-b8fc-4eb7121a4765 nodeName:}" failed. No retries permitted until 2026-11-02 21:24:48.968569689 +0000 UTC m=+714.944867447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls") pod "maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" (UID: "110c5f36-3efa-4fdb-b8fc-4eb7121a4765") : secret "maas-api-e2e-worker-w1-c2dd3b-serving-cert" not found Nov 02 21:24:48.469023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.469001 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.470666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.470647 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-metrics-certs\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.488335 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.488288 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" podStartSLOduration=1.488277246 podStartE2EDuration="1.488277246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:48.486915194 +0000 UTC m=+714.463212963" watchObservedRunningTime="2026-11-02 21:24:48.488277246 +0000 UTC m=+714.464575064" Nov 02 21:24:48.521316 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.521262 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldpwv\" (UniqueName: \"kubernetes.io/projected/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-kube-api-access-ldpwv\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.973859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.973828 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:48.976022 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:48.975994 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:49.188966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:49.188929 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:49.564639 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:49.564560 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw"] Nov 02 21:24:49.571721 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:24:49.571688 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110c5f36_3efa_4fdb_b8fc_4eb7121a4765.slice/crio-e6396272a2288a566657c6c8f83cbeff95e6100455118217cceb2dd5a71662fc WatchSource:0}: Error finding container e6396272a2288a566657c6c8f83cbeff95e6100455118217cceb2dd5a71662fc: Status 404 returned error can't find the container with id e6396272a2288a566657c6c8f83cbeff95e6100455118217cceb2dd5a71662fc Nov 02 21:24:50.316804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:50.316773 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 21:24:50.379773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:50.379743 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" event={"ID":"110c5f36-3efa-4fdb-b8fc-4eb7121a4765","Type":"ContainerStarted","Data":"e6396272a2288a566657c6c8f83cbeff95e6100455118217cceb2dd5a71662fc"} Nov 02 21:24:51.386336 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:51.386298 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" event={"ID":"110c5f36-3efa-4fdb-b8fc-4eb7121a4765","Type":"ContainerStarted","Data":"312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8"} Nov 02 21:24:51.386733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:51.386476 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:51.491722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:51.491676 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" podStartSLOduration=2.549546705 podStartE2EDuration="3.491662246s" totalImagesPullingTime="942.115541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:51.490989854 +0000 UTC m=+717.467287629" watchObservedRunningTime="2026-11-02 21:24:51.491662246 +0000 UTC m=+717.467960021" Nov 02 21:24:52.367110 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:52.367081 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 21:24:54.763231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:54.763198 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm"] Nov 02 21:24:54.770201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:54.770174 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:54.820108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:54.820038 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm"] Nov 02 21:24:54.863531 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:54.863492 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-dd6574-dockercfg-z8jd9\"" Nov 02 21:24:54.880474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:54.880455 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-dd6574\"" Nov 02 21:24:54.933515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:54.933483 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:54.933646 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:54.933543 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vp42\" (UniqueName: \"kubernetes.io/projected/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-kube-api-access-8vp42\") pod \"payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:55.034706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:55.034634 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:55.034706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:55.034680 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vp42\" (UniqueName: \"kubernetes.io/projected/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-kube-api-access-8vp42\") pod \"payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:55.035361 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:55.035340 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:55.087415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:55.087383 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vp42\" (UniqueName: \"kubernetes.io/projected/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-kube-api-access-8vp42\") pod \"payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:55.344536 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:55.344452 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:24:55.386799 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:55.386772 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:55.754939 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:55.754908 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm"] Nov 02 21:24:56.220082 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:56.220053 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:24:56.414789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:56.414759 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" event={"ID":"cfaa81dc-d407-4a1e-b9ff-950e1396ab81","Type":"ContainerStarted","Data":"99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11"} Nov 02 21:24:56.414949 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:56.414807 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" event={"ID":"cfaa81dc-d407-4a1e-b9ff-950e1396ab81","Type":"ContainerStarted","Data":"9d6034f8739ffa28ab41dbdbfa9cabc07a3cd113637e1c79f9dbad39eaecc04f"} Nov 02 21:24:56.414949 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:56.414868 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:24:56.520629 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:56.520537 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" podStartSLOduration=2.5205078690000002 podStartE2EDuration="2.520507869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:24:56.518395799 +0000 UTC m=+722.494693571" watchObservedRunningTime="2026-11-02 21:24:56.520507869 +0000 UTC m=+722.496805644" Nov 02 21:24:57.397708 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:57.397678 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 21:24:59.376668 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:24:59.376641 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 21:25:02.350508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.350474 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r"] Nov 02 21:25:02.355727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.355706 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.408194 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.408159 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r"] Nov 02 21:25:02.451183 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.451154 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4e2b27-serving-cert\"" Nov 02 21:25:02.510016 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.509980 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-metrics-certs\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.510196 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.510021 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.510269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.510216 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6mv\" (UniqueName: \"kubernetes.io/projected/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-kube-api-access-fr6mv\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.510325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.510309 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.612260 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.612171 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.612419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.612288 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-metrics-certs\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.612419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.612320 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.612419 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:25:02.612333 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-serving-cert: secret "maas-api-e2e-worker-w4-4e2b27-serving-cert" not found Nov 02 21:25:02.612419 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:25:02.612411 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls podName:5bddd2c8-5804-4ae2-a11d-c708cbc56e59 nodeName:}" failed. No retries permitted until 2026-11-02 21:25:03.112390794 +0000 UTC m=+729.088688553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls") pod "maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" (UID: "5bddd2c8-5804-4ae2-a11d-c708cbc56e59") : secret "maas-api-e2e-worker-w4-4e2b27-serving-cert" not found Nov 02 21:25:02.612612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.612431 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6mv\" (UniqueName: \"kubernetes.io/projected/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-kube-api-access-fr6mv\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.613027 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.613006 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.614997 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.614970 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-metrics-certs\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:02.665509 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:02.665477 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6mv\" (UniqueName: \"kubernetes.io/projected/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-kube-api-access-fr6mv\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:03.118854 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:03.118813 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:03.121208 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:03.121179 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:03.269900 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:03.269862 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:03.640618 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:03.640585 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r"] Nov 02 21:25:03.648224 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:25:03.648188 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bddd2c8_5804_4ae2_a11d_c708cbc56e59.slice/crio-344484c5cb37d4715accce8f5791baf846362ac3d24a56933ddc3ae52946d795 WatchSource:0}: Error finding container 344484c5cb37d4715accce8f5791baf846362ac3d24a56933ddc3ae52946d795: Status 404 returned error can't find the container with id 344484c5cb37d4715accce8f5791baf846362ac3d24a56933ddc3ae52946d795 Nov 02 21:25:04.457131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:04.457092 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" event={"ID":"5bddd2c8-5804-4ae2-a11d-c708cbc56e59","Type":"ContainerStarted","Data":"344484c5cb37d4715accce8f5791baf846362ac3d24a56933ddc3ae52946d795"} Nov 02 21:25:05.464394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:05.464360 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" event={"ID":"5bddd2c8-5804-4ae2-a11d-c708cbc56e59","Type":"ContainerStarted","Data":"4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd"} Nov 02 21:25:05.464813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:05.464467 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:05.570992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:05.570920 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" podStartSLOduration=2.685970673 podStartE2EDuration="3.570903946s" totalImagesPullingTime="884.933273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:05.569693077 +0000 UTC m=+731.545990862" watchObservedRunningTime="2026-11-02 21:25:05.570903946 +0000 UTC m=+731.547201722" Nov 02 21:25:07.424465 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:07.424434 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:25:08.116309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:08.116272 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:25:11.479144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.478402 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 21:25:11.591953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.591918 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4"] Nov 02 21:25:11.596041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.596016 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.606745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.606722 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6296d\" (UniqueName: \"kubernetes.io/projected/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-kube-api-access-6296d\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.606890 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.606864 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.606959 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.606944 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.606998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.606977 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-metrics-certs\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.677547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.651075 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-9a6cf2-serving-cert\"" Nov 02 21:25:11.682382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.682353 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4"] Nov 02 21:25:11.711593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.710964 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.711593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.711019 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.711593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.711045 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-metrics-certs\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.711593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.711099 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6296d\" (UniqueName: \"kubernetes.io/projected/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-kube-api-access-6296d\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.711992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.711979 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.712082 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:25:11.712064 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-serving-cert: secret "maas-api-e2e-worker-w3-9a6cf2-serving-cert" not found Nov 02 21:25:11.712235 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:25:11.712226 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls podName:9c0380e3-4e17-4b1e-aa4d-42d596aa8425 nodeName:}" failed. No retries permitted until 2026-11-02 21:25:12.212212839 +0000 UTC m=+738.188510592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls") pod "maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" (UID: "9c0380e3-4e17-4b1e-aa4d-42d596aa8425") : secret "maas-api-e2e-worker-w3-9a6cf2-serving-cert" not found Nov 02 21:25:11.714934 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.714918 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-metrics-certs\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:11.780000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:11.779923 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6296d\" (UniqueName: \"kubernetes.io/projected/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-kube-api-access-6296d\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:12.217776 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:12.217735 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:12.222399 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:12.222373 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:12.508988 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:12.508902 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:12.971554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:12.957553 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4"] Nov 02 21:25:13.508294 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:13.508267 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" event={"ID":"9c0380e3-4e17-4b1e-aa4d-42d596aa8425","Type":"ContainerStarted","Data":"7459269d7ac978fb1c10250ba014663ade8604783408bda043d9eab85d5571cb"} Nov 02 21:25:14.516646 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:14.516547 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" event={"ID":"9c0380e3-4e17-4b1e-aa4d-42d596aa8425","Type":"ContainerStarted","Data":"6802fc78f1084634a5a0b1bbea7d3d003431624a79586b0ba6d3c0072b4e13b9"} Nov 02 21:25:14.516646 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:14.516640 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:14.623367 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:14.623309 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" podStartSLOduration=2.81304436 podStartE2EDuration="3.623295555s" totalImagesPullingTime="810.251195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:25:14.622042677 +0000 UTC m=+740.598340451" watchObservedRunningTime="2026-11-02 21:25:14.623295555 +0000 UTC m=+740.599593330" Nov 02 21:25:15.553262 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:25:15.553222 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:25:17.305703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.305663 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:25:17.306778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.305663 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:25:17.306778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.305748 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:25:17.306778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.305764 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:25:17.537295 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.537221 4510 generic.go:356] "Generic (PLEG): container finished" podID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerID="d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7" exitCode=0 Nov 02 21:25:17.537458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.537296 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" event={"ID":"0a9f15c8-aea4-4c05-9945-301b760e5fb7","Type":"ContainerDied","Data":"d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7"} Nov 02 21:25:17.537737 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.537718 4510 scope.go:122] "RemoveContainer" containerID="d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7" Nov 02 21:25:17.539059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.539036 4510 generic.go:356] "Generic (PLEG): container finished" podID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerID="7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42" exitCode=0 Nov 02 21:25:17.539140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.539099 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" event={"ID":"7b68b66b-eb3b-4eff-8e07-c36076d211ee","Type":"ContainerDied","Data":"7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42"} Nov 02 21:25:17.539242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.539221 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:17.539294 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.539282 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:17.539328 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.539315 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:17.539403 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.539391 4510 scope.go:122] "RemoveContainer" containerID="7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42" Nov 02 21:25:17.540550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.540512 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:17.540626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.540586 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:17.540667 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:17.540631 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:18.583292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:18.583261 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" event={"ID":"7b68b66b-eb3b-4eff-8e07-c36076d211ee","Type":"ContainerStarted","Data":"db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9"} Nov 02 21:25:18.585653 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:18.585625 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" event={"ID":"0a9f15c8-aea4-4c05-9945-301b760e5fb7","Type":"ContainerStarted","Data":"c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee"} Nov 02 21:25:19.074544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:19.074105 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:25:19.074760 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:19.074717 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:25:19.074827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:19.074776 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 21:25:19.367543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:19.365958 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:25:19.367543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:19.366078 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 21:25:19.367543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:19.366123 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 21:25:20.079545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:20.079000 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:25:20.079545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:20.079067 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 21:25:20.370547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:20.366987 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 21:25:20.370547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:20.367052 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 21:25:20.554721 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:20.554687 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 21:25:21.074544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:21.074115 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:25:21.074544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:21.074196 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 21:25:21.379542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:21.377964 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 21:25:21.379542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:21.378040 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 21:25:22.075551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:22.075375 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:25:22.075551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:22.075444 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 21:25:22.377546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:22.375867 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 21:25:22.377546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:22.375930 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 21:25:23.075004 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:23.074964 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 21:25:23.075622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:23.075588 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 21:25:23.367037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:23.367002 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 21:25:23.367297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:23.367269 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 21:25:24.158959 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:24.158925 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:25:24.159338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:24.159288 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:25:24.160062 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:24.160044 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:25:24.368897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:24.368869 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:25:24.369159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:24.369128 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:25:24.369879 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:24.369863 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:25:29.681308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:29.681274 4510 generic.go:356] "Generic (PLEG): container finished" podID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerID="2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3" exitCode=0 Nov 02 21:25:29.681873 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:29.681352 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" event={"ID":"b023881e-83cb-44b7-9db4-246dd5665b6a","Type":"ContainerDied","Data":"2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3"} Nov 02 21:25:29.681948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:29.681899 4510 scope.go:122] "RemoveContainer" containerID="2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3" Nov 02 21:25:29.683226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:29.683203 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:29.683284 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:29.683268 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:29.683319 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:29.683300 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:25:30.689320 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:30.689285 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" event={"ID":"b023881e-83cb-44b7-9db4-246dd5665b6a","Type":"ContainerStarted","Data":"e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744"} Nov 02 21:25:31.264834 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:31.264798 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:25:31.494793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:31.494762 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:25:31.495899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:31.495877 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:25:31.495961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:31.495935 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 21:25:32.495290 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:32.495254 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:25:32.495731 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:32.495307 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 21:25:32.751247 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:32.751158 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:25:33.494698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:33.494663 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:25:33.494874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:33.494714 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 21:25:34.494858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:34.494824 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:25:34.495254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:34.494876 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 21:25:35.495260 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:35.495223 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 21:25:35.495686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:35.495276 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 21:25:36.499333 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:36.499304 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:25:36.500144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:36.500127 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 21:25:45.532211 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:25:45.532174 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:25:54.601717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:25:54.601683 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:26:09.563590 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:26:09.563548 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:26:15.531055 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:26:15.531017 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:26:30.473334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:26:30.473290 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:26:45.532438 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:26:45.532402 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:27:00.515754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:27:00.515721 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:27:15.530702 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:27:15.530625 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:27:22.726622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:27:22.726591 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:27:45.531877 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:27:45.531835 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:27:49.322819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:27:49.322780 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:28:15.529463 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:28:15.529428 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:28:18.323138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:18.323102 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:28:31.212161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.212126 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7"] Nov 02 21:28:31.215982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.215966 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.265492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.265464 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-dd6574-openshift-default-dockercfg-wcc2w\"" Nov 02 21:28:31.268965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.268924 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgh2\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-kube-api-access-zvgh2\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269040 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269075 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269146 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269261 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269248 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269328 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269313 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab97c6c4-9002-4974-af0b-5a88740a614b-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269484 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269568 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.269622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.269601 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.315407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.315383 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7"] Nov 02 21:28:31.371240 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371205 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371240 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371243 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371283 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371319 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgh2\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-kube-api-access-zvgh2\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371360 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371400 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371425 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371502 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371602 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab97c6c4-9002-4974-af0b-5a88740a614b-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371665 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371760 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371788 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.371894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.371845 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.372343 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.372320 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab97c6c4-9002-4974-af0b-5a88740a614b-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.373778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.373753 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.373906 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.373889 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.424307 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.424273 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.424477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.424375 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgh2\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-kube-api-access-zvgh2\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.528543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.528454 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:31.955696 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.955658 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7"] Nov 02 21:28:31.962509 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:28:31.962483 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab97c6c4_9002_4974_af0b_5a88740a614b.slice/crio-10d38a84ccef4c3442f9f42ccbc4de356fd4703bc9c0bcc873db333a85581147 WatchSource:0}: Error finding container 10d38a84ccef4c3442f9f42ccbc4de356fd4703bc9c0bcc873db333a85581147: Status 404 returned error can't find the container with id 10d38a84ccef4c3442f9f42ccbc4de356fd4703bc9c0bcc873db333a85581147 Nov 02 21:28:31.964783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.964752 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:31.964869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.964816 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:31.964869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:31.964863 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:32.680142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:32.680104 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" event={"ID":"ab97c6c4-9002-4974-af0b-5a88740a614b","Type":"ContainerStarted","Data":"ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f"} Nov 02 21:28:32.680572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:32.680162 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" event={"ID":"ab97c6c4-9002-4974-af0b-5a88740a614b","Type":"ContainerStarted","Data":"10d38a84ccef4c3442f9f42ccbc4de356fd4703bc9c0bcc873db333a85581147"} Nov 02 21:28:32.892189 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:32.892137 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" podStartSLOduration=1.892122068 podStartE2EDuration="1.892122068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:32.846054904 +0000 UTC m=+938.822352737" watchObservedRunningTime="2026-11-02 21:28:32.892122068 +0000 UTC m=+938.868419844" Nov 02 21:28:33.540543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:33.537927 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:33.551677 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:33.551642 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:28:33.551942 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:33.551905 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:28:34.543545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:34.542866 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:28:34.543545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:34.542943 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:28:35.541599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:35.530900 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:28:35.541599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:35.530962 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:28:36.535939 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:36.535899 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:28:36.536691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:36.536660 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:28:37.538545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:37.532895 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 02 21:28:37.538545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:37.532960 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 02 21:28:38.578805 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:38.578767 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:38.579844 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:38.579762 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:38.580554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:38.580512 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:28:45.446891 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.446851 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw"] Nov 02 21:28:45.451433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.451411 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.499794 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.499763 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ef6b43-openshift-default-dockercfg-2pk6b\"" Nov 02 21:28:45.532224 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:28:45.532190 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:28:45.549077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.549054 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw"] Nov 02 21:28:45.641218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641183 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641393 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641223 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641393 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641252 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641420 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcfl\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-kube-api-access-mjcfl\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641466 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641640 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641600 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641696 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641637 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641774 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641753 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.641836 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.641787 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.742567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.742458 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.742567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.742555 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.742800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.742575 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.742800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.742599 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.742913 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.742815 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcfl\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-kube-api-access-mjcfl\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.742974 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.742919 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743019 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743055 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743154 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743210 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743285 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743608 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743585 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743755 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.743857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.743835 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.745049 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.745026 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.745285 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.745267 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.795136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.795113 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:45.795440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:45.795409 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcfl\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-kube-api-access-mjcfl\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:46.066309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.066228 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:46.492982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.492948 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw"] Nov 02 21:28:46.499927 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:28:46.499901 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26a9ed3_7950_4ad2_957d_0deb5beb99c0.slice/crio-fd4881a49ac62a610e993d475560f14d232b8e2e3c449ad152057a8f9dbf8fdf WatchSource:0}: Error finding container fd4881a49ac62a610e993d475560f14d232b8e2e3c449ad152057a8f9dbf8fdf: Status 404 returned error can't find the container with id fd4881a49ac62a610e993d475560f14d232b8e2e3c449ad152057a8f9dbf8fdf Nov 02 21:28:46.502130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.502088 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:46.502201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.502163 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:46.502201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.502193 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:28:46.758368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.758281 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" event={"ID":"f26a9ed3-7950-4ad2-957d-0deb5beb99c0","Type":"ContainerStarted","Data":"61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69"} Nov 02 21:28:46.758368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.758343 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" event={"ID":"f26a9ed3-7950-4ad2-957d-0deb5beb99c0","Type":"ContainerStarted","Data":"fd4881a49ac62a610e993d475560f14d232b8e2e3c449ad152057a8f9dbf8fdf"} Nov 02 21:28:46.865741 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:46.865693 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podStartSLOduration=1.865679466 podStartE2EDuration="1.865679466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:28:46.865182963 +0000 UTC m=+952.841480773" watchObservedRunningTime="2026-11-02 21:28:46.865679466 +0000 UTC m=+952.841977250" Nov 02 21:28:47.067260 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:47.067170 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:47.068877 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:47.068853 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:28:47.069001 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:47.068898 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 21:28:48.067613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:48.067503 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:28:48.067613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:48.067593 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 21:28:49.067105 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:49.067068 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:28:49.067324 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:49.067125 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 21:28:50.067541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:50.067475 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:28:50.067969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:50.067560 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 21:28:51.067283 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:51.067248 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:28:51.067465 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:51.067302 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 21:28:52.067104 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:52.067067 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 02 21:28:52.067554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:52.067123 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 02 21:28:53.149452 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:53.149418 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:53.150224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:53.150196 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:28:53.150966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:28:53.150946 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:29:07.232626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:07.232591 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:29:15.530162 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:29:15.530128 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:29:45.039584 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.039554 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm"] Nov 02 21:29:45.043440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.043419 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.091585 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.091562 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c2dd3b-openshift-default-dockercfg-rvfbh\"" Nov 02 21:29:45.109894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.109866 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.109921 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.110034 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.110078 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110181 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.110163 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110220 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.110198 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110275 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.110219 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110320 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.110297 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8kcm\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-kube-api-access-d8kcm\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.110358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.110329 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.140999 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.140974 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm"] Nov 02 21:29:45.211392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211349 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211413 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211469 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211512 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211586 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211625 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8kcm\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-kube-api-access-d8kcm\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211675 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211808 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.211883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211820 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.212087 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.211894 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.212149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.212132 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.212243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.212223 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.212454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.212434 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.212560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.212500 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.214373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.214348 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.214740 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.214717 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.267147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.267117 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.267437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.267414 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8kcm\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-kube-api-access-d8kcm\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.356796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.356728 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:45.534952 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:29:45.534914 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:29:45.792592 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.792553 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm"] Nov 02 21:29:45.800968 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:29:45.800939 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02178fd7_d0a9_4ee0_ad2c_fa98bc6ddeed.slice/crio-4fdd1a8e5aa3e85adcd56ba39351c4d78dbf78fb8e71d47329451388a1001828 WatchSource:0}: Error finding container 4fdd1a8e5aa3e85adcd56ba39351c4d78dbf78fb8e71d47329451388a1001828: Status 404 returned error can't find the container with id 4fdd1a8e5aa3e85adcd56ba39351c4d78dbf78fb8e71d47329451388a1001828 Nov 02 21:29:45.803916 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.803881 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:45.804001 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.803962 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:45.804039 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:45.804003 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:29:46.129585 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:46.129548 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" event={"ID":"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed","Type":"ContainerStarted","Data":"7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc"} Nov 02 21:29:46.130065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:46.129613 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" event={"ID":"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed","Type":"ContainerStarted","Data":"4fdd1a8e5aa3e85adcd56ba39351c4d78dbf78fb8e71d47329451388a1001828"} Nov 02 21:29:46.238185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:46.238135 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podStartSLOduration=2.238119115 podStartE2EDuration="2.238119115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:29:46.237082228 +0000 UTC m=+1012.213380016" watchObservedRunningTime="2026-11-02 21:29:46.238119115 +0000 UTC m=+1012.214416922" Nov 02 21:29:46.357912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:46.357869 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:46.359239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:46.359211 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 21:29:46.359373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:46.359263 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 21:29:47.357332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:47.357302 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 21:29:47.357746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:47.357352 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 21:29:48.357165 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:48.357124 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 21:29:48.357364 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:48.357182 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 21:29:49.357425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:49.357389 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 21:29:49.357869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:49.357443 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 21:29:50.357100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:50.357063 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 21:29:50.357269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:50.357121 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 21:29:51.357335 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:51.357298 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 02 21:29:51.357766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:51.357350 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 02 21:29:52.461960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:52.461929 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:52.462663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:52.462640 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:52.463430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:52.463411 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:29:53.152044 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:29:53.152013 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:30:00.164868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.164834 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw"] Nov 02 21:30:00.169935 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.169915 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.185990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.185968 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9"] Nov 02 21:30:00.189665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.189647 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.217825 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.217796 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 21:30:00.238883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.238863 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrm62\" (UniqueName: \"kubernetes.io/projected/1f975752-56ae-4b16-9f14-d122c358139e-kube-api-access-mrm62\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.238974 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.238928 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f975752-56ae-4b16-9f14-d122c358139e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.262243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.262131 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw"] Nov 02 21:30:00.308116 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.308084 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9"] Nov 02 21:30:00.340477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.340451 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk65m\" (UniqueName: \"kubernetes.io/projected/da4e8889-10b0-4a81-84c0-dc2332cd082a-kube-api-access-jk65m\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.340671 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.340648 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrm62\" (UniqueName: \"kubernetes.io/projected/1f975752-56ae-4b16-9f14-d122c358139e-kube-api-access-mrm62\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.340724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.340691 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4e8889-10b0-4a81-84c0-dc2332cd082a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.340766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.340747 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f975752-56ae-4b16-9f14-d122c358139e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.341352 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.341336 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f975752-56ae-4b16-9f14-d122c358139e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.392932 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.392908 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrm62\" (UniqueName: \"kubernetes.io/projected/1f975752-56ae-4b16-9f14-d122c358139e-kube-api-access-mrm62\") pod \"maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.442168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.442095 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4e8889-10b0-4a81-84c0-dc2332cd082a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.442305 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.442192 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk65m\" (UniqueName: \"kubernetes.io/projected/da4e8889-10b0-4a81-84c0-dc2332cd082a-kube-api-access-jk65m\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.442690 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.442672 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4e8889-10b0-4a81-84c0-dc2332cd082a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.483081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.483057 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:00.495079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.495053 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk65m\" (UniqueName: \"kubernetes.io/projected/da4e8889-10b0-4a81-84c0-dc2332cd082a-kube-api-access-jk65m\") pod \"maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.502738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.502718 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:00.857190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.857114 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw"] Nov 02 21:30:00.865167 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:30:00.865133 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f975752_56ae_4b16_9f14_d122c358139e.slice/crio-b44be010481fc7081042b47c694938db757029f023e84034cdd11f6661ba792f WatchSource:0}: Error finding container b44be010481fc7081042b47c694938db757029f023e84034cdd11f6661ba792f: Status 404 returned error can't find the container with id b44be010481fc7081042b47c694938db757029f023e84034cdd11f6661ba792f Nov 02 21:30:00.867273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.867249 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:30:00.908063 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:00.908036 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9"] Nov 02 21:30:00.916395 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:30:00.916371 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4e8889_10b0_4a81_84c0_dc2332cd082a.slice/crio-8b4b7158837a2257727ac9a20d781e86e7932d4f8742805f6386359fb485c5a5 WatchSource:0}: Error finding container 8b4b7158837a2257727ac9a20d781e86e7932d4f8742805f6386359fb485c5a5: Status 404 returned error can't find the container with id 8b4b7158837a2257727ac9a20d781e86e7932d4f8742805f6386359fb485c5a5 Nov 02 21:30:01.251400 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:01.251361 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" event={"ID":"1f975752-56ae-4b16-9f14-d122c358139e","Type":"ContainerStarted","Data":"b44be010481fc7081042b47c694938db757029f023e84034cdd11f6661ba792f"} Nov 02 21:30:01.252535 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:01.252498 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" event={"ID":"da4e8889-10b0-4a81-84c0-dc2332cd082a","Type":"ContainerStarted","Data":"8b4b7158837a2257727ac9a20d781e86e7932d4f8742805f6386359fb485c5a5"} Nov 02 21:30:10.307926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:10.307836 4510 generic.go:356] "Generic (PLEG): container finished" podID="1f975752-56ae-4b16-9f14-d122c358139e" containerID="df6cc2175883f3cc3ab662d0b37de9f4f92b86e5ce526129e0ff4c93148999ba" exitCode=60 Nov 02 21:30:10.308377 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:10.307914 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" event={"ID":"1f975752-56ae-4b16-9f14-d122c358139e","Type":"ContainerDied","Data":"df6cc2175883f3cc3ab662d0b37de9f4f92b86e5ce526129e0ff4c93148999ba"} Nov 02 21:30:10.308377 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:10.308203 4510 scope.go:122] "RemoveContainer" containerID="df6cc2175883f3cc3ab662d0b37de9f4f92b86e5ce526129e0ff4c93148999ba" Nov 02 21:30:10.309771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:10.309737 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" event={"ID":"da4e8889-10b0-4a81-84c0-dc2332cd082a","Type":"ContainerStarted","Data":"ecf7914c979d50af1f4b4bbd203e6ba9c1887d5182a9e86a0840f8cef345673d"} Nov 02 21:30:10.517714 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:10.517620 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" podStartSLOduration=1.930886225 podStartE2EDuration="10.517603412s" totalImagesPullingTime="8.586717187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:10.515836714 +0000 UTC m=+1036.492134489" watchObservedRunningTime="2026-11-02 21:30:10.517603412 +0000 UTC m=+1036.493901188" Nov 02 21:30:10.550567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:10.550532 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282"] Nov 02 21:30:10.550767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:10.550745 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" containerID="cri-o://d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded" gracePeriod=30 Nov 02 21:30:11.317929 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:11.317903 4510 generic.go:356] "Generic (PLEG): container finished" podID="1f975752-56ae-4b16-9f14-d122c358139e" containerID="6b60704b85c9232e7521e8c03a65797b1e214751819f9e288591c476cadf8076" exitCode=60 Nov 02 21:30:11.318309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:11.317984 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" event={"ID":"1f975752-56ae-4b16-9f14-d122c358139e","Type":"ContainerDied","Data":"6b60704b85c9232e7521e8c03a65797b1e214751819f9e288591c476cadf8076"} Nov 02 21:30:11.318309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:11.318046 4510 scope.go:122] "RemoveContainer" containerID="df6cc2175883f3cc3ab662d0b37de9f4f92b86e5ce526129e0ff4c93148999ba" Nov 02 21:30:11.318309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:11.318199 4510 scope.go:122] "RemoveContainer" containerID="6b60704b85c9232e7521e8c03a65797b1e214751819f9e288591c476cadf8076" Nov 02 21:30:11.318457 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:11.318439 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw_odh-ai-gateway-infra(1f975752-56ae-4b16-9f14-d122c358139e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" podUID="1f975752-56ae-4b16-9f14-d122c358139e" Nov 02 21:30:11.320894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:11.320207 4510 generic.go:356] "Generic (PLEG): container finished" podID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerID="ecf7914c979d50af1f4b4bbd203e6ba9c1887d5182a9e86a0840f8cef345673d" exitCode=60 Nov 02 21:30:11.320894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:11.320254 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" event={"ID":"da4e8889-10b0-4a81-84c0-dc2332cd082a","Type":"ContainerDied","Data":"ecf7914c979d50af1f4b4bbd203e6ba9c1887d5182a9e86a0840f8cef345673d"} Nov 02 21:30:11.320894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:11.320580 4510 scope.go:122] "RemoveContainer" containerID="ecf7914c979d50af1f4b4bbd203e6ba9c1887d5182a9e86a0840f8cef345673d" Nov 02 21:30:12.328534 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:12.328495 4510 generic.go:356] "Generic (PLEG): container finished" podID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerID="5c2720a50b37d8db64000eb5f43a2234f723b482b95947b521a072e12353f2f6" exitCode=60 Nov 02 21:30:12.328938 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:12.328584 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" event={"ID":"da4e8889-10b0-4a81-84c0-dc2332cd082a","Type":"ContainerDied","Data":"5c2720a50b37d8db64000eb5f43a2234f723b482b95947b521a072e12353f2f6"} Nov 02 21:30:12.328938 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:12.328649 4510 scope.go:122] "RemoveContainer" containerID="ecf7914c979d50af1f4b4bbd203e6ba9c1887d5182a9e86a0840f8cef345673d" Nov 02 21:30:12.328938 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:12.328915 4510 scope.go:122] "RemoveContainer" containerID="5c2720a50b37d8db64000eb5f43a2234f723b482b95947b521a072e12353f2f6" Nov 02 21:30:12.329217 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:12.329195 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9_odh-ai-gateway-infra(da4e8889-10b0-4a81-84c0-dc2332cd082a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" Nov 02 21:30:12.330616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:12.330601 4510 scope.go:122] "RemoveContainer" containerID="6b60704b85c9232e7521e8c03a65797b1e214751819f9e288591c476cadf8076" Nov 02 21:30:12.330839 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:12.330822 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw_odh-ai-gateway-infra(1f975752-56ae-4b16-9f14-d122c358139e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" podUID="1f975752-56ae-4b16-9f14-d122c358139e" Nov 02 21:30:13.337419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:13.337392 4510 scope.go:122] "RemoveContainer" containerID="5c2720a50b37d8db64000eb5f43a2234f723b482b95947b521a072e12353f2f6" Nov 02 21:30:13.337840 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:13.337674 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9_odh-ai-gateway-infra(da4e8889-10b0-4a81-84c0-dc2332cd082a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" Nov 02 21:30:15.530962 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:15.530929 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:30:15.849756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.849734 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:30:15.922830 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.922795 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-socket\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.922868 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-token\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.922900 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-podinfo\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.922930 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-credential-socket\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.922963 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-envoy\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923045 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-socket" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:15.923229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923136 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istiod-ca-cert\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923168 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-data\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923208 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-kube-api-access-jc6z9\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-kube-api-access-jc6z9\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923426 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923259 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-certs\") pod \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\" (UID: \"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c\") " Nov 02 21:30:15.923511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923483 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-data" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:15.923648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923628 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-credential-socket" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:15.923710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923671 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-certs" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:15.923856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923830 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.923856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923857 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.924050 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923855 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istiod-ca-cert" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:15.924050 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923873 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.924050 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.923887 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:15.925754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.925729 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-envoy" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:15.925879 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.925778 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-token" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:15.926210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.926189 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-podinfo" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:30:15.926320 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:15.926212 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-kube-api-access-jc6z9" pod "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" (UID: "312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c"). InnerVolumeSpecName "kube-api-access-jc6z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:16.025327 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.025292 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc6z9\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-kube-api-access-jc6z9\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:16.025327 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.025327 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:16.025557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.025340 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:16.025557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.025353 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:16.025557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.025365 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:16.359511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.359466 4510 generic.go:356] "Generic (PLEG): container finished" podID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerID="d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded" exitCode=0 Nov 02 21:30:16.359722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.359554 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" Nov 02 21:30:16.359722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.359553 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" event={"ID":"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c","Type":"ContainerDied","Data":"d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded"} Nov 02 21:30:16.359722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.359627 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282" event={"ID":"312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c","Type":"ContainerDied","Data":"b1451650f73755833a7bb065d2abffad2a38924fb163910ef53b0c3fbd7192de"} Nov 02 21:30:16.359722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.359660 4510 scope.go:122] "RemoveContainer" containerID="d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded" Nov 02 21:30:16.372781 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.372764 4510 scope.go:122] "RemoveContainer" containerID="d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded" Nov 02 21:30:16.373103 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:16.373075 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded\": container with ID starting with d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded not found: ID does not exist" containerID="d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded" Nov 02 21:30:16.373199 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.373116 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded"} err="failed to get container status \"d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded\": rpc error: code = NotFound desc = could not find container \"d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded\": container with ID starting with d3f32545714ac815b7d65cc03bfb536b0acc3628bbbae233c8965a8672ac1ded not found: ID does not exist" Nov 02 21:30:16.521319 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.521228 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282"] Nov 02 21:30:16.567333 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:16.567297 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7x282"] Nov 02 21:30:18.153361 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:18.153320 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" path="/var/lib/kubelet/pods/312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c/volumes" Nov 02 21:30:25.147016 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:25.146984 4510 scope.go:122] "RemoveContainer" containerID="6b60704b85c9232e7521e8c03a65797b1e214751819f9e288591c476cadf8076" Nov 02 21:30:25.586841 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:25.586806 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9"] Nov 02 21:30:25.587021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:25.586999 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" containerID="cri-o://694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6" gracePeriod=30 Nov 02 21:30:26.146743 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:26.146708 4510 scope.go:122] "RemoveContainer" containerID="5c2720a50b37d8db64000eb5f43a2234f723b482b95947b521a072e12353f2f6" Nov 02 21:30:26.424377 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:26.424348 4510 generic.go:356] "Generic (PLEG): container finished" podID="1f975752-56ae-4b16-9f14-d122c358139e" containerID="099c14b574a05fb06849f7b50c089ecc12cc5da43c12e4fa9f57e2b0616cca30" exitCode=60 Nov 02 21:30:26.424789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:26.424419 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" event={"ID":"1f975752-56ae-4b16-9f14-d122c358139e","Type":"ContainerDied","Data":"099c14b574a05fb06849f7b50c089ecc12cc5da43c12e4fa9f57e2b0616cca30"} Nov 02 21:30:26.424789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:26.424473 4510 scope.go:122] "RemoveContainer" containerID="6b60704b85c9232e7521e8c03a65797b1e214751819f9e288591c476cadf8076" Nov 02 21:30:26.424955 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:26.424791 4510 scope.go:122] "RemoveContainer" containerID="099c14b574a05fb06849f7b50c089ecc12cc5da43c12e4fa9f57e2b0616cca30" Nov 02 21:30:26.425084 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:26.425064 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw_odh-ai-gateway-infra(1f975752-56ae-4b16-9f14-d122c358139e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" podUID="1f975752-56ae-4b16-9f14-d122c358139e" Nov 02 21:30:26.426631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:26.426612 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" event={"ID":"da4e8889-10b0-4a81-84c0-dc2332cd082a","Type":"ContainerStarted","Data":"88bb076f30418d3fb227cc7e295663c597cca1884a2b223920c842a2bedce348"} Nov 02 21:30:27.442435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.442404 4510 generic.go:356] "Generic (PLEG): container finished" podID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerID="88bb076f30418d3fb227cc7e295663c597cca1884a2b223920c842a2bedce348" exitCode=60 Nov 02 21:30:27.442894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.442441 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" event={"ID":"da4e8889-10b0-4a81-84c0-dc2332cd082a","Type":"ContainerDied","Data":"88bb076f30418d3fb227cc7e295663c597cca1884a2b223920c842a2bedce348"} Nov 02 21:30:27.442894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.442505 4510 scope.go:122] "RemoveContainer" containerID="5c2720a50b37d8db64000eb5f43a2234f723b482b95947b521a072e12353f2f6" Nov 02 21:30:27.442894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.442813 4510 scope.go:122] "RemoveContainer" containerID="88bb076f30418d3fb227cc7e295663c597cca1884a2b223920c842a2bedce348" Nov 02 21:30:27.443095 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:27.443070 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9_odh-ai-gateway-infra(da4e8889-10b0-4a81-84c0-dc2332cd082a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" Nov 02 21:30:27.540252 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.540219 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw"] Nov 02 21:30:27.643016 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.642976 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9"] Nov 02 21:30:27.701759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.701700 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:27.867780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.867750 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f975752-56ae-4b16-9f14-d122c358139e-kube-api-access-mrm62\" (UniqueName: \"kubernetes.io/projected/1f975752-56ae-4b16-9f14-d122c358139e-kube-api-access-mrm62\") pod \"1f975752-56ae-4b16-9f14-d122c358139e\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " Nov 02 21:30:27.867934 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.867807 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f975752-56ae-4b16-9f14-d122c358139e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f975752-56ae-4b16-9f14-d122c358139e-maas-api-service-ca\") pod \"1f975752-56ae-4b16-9f14-d122c358139e\" (UID: \"1f975752-56ae-4b16-9f14-d122c358139e\") " Nov 02 21:30:27.868227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.868197 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f975752-56ae-4b16-9f14-d122c358139e-maas-api-service-ca" pod "1f975752-56ae-4b16-9f14-d122c358139e" (UID: "1f975752-56ae-4b16-9f14-d122c358139e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:27.869701 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.869670 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f975752-56ae-4b16-9f14-d122c358139e-kube-api-access-mrm62" pod "1f975752-56ae-4b16-9f14-d122c358139e" (UID: "1f975752-56ae-4b16-9f14-d122c358139e"). InnerVolumeSpecName "kube-api-access-mrm62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:27.969440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.969359 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrm62\" (UniqueName: \"kubernetes.io/projected/1f975752-56ae-4b16-9f14-d122c358139e-kube-api-access-mrm62\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:27.969440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:27.969392 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f975752-56ae-4b16-9f14-d122c358139e-maas-api-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.450816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.450788 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" event={"ID":"1f975752-56ae-4b16-9f14-d122c358139e","Type":"ContainerDied","Data":"b44be010481fc7081042b47c694938db757029f023e84034cdd11f6661ba792f"} Nov 02 21:30:28.451209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.450817 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw" Nov 02 21:30:28.451209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.450852 4510 scope.go:122] "RemoveContainer" containerID="099c14b574a05fb06849f7b50c089ecc12cc5da43c12e4fa9f57e2b0616cca30" Nov 02 21:30:28.585761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.585731 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:28.657317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.657286 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw"] Nov 02 21:30:28.659683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.659661 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9a6cf2-29894250-hj2cw"] Nov 02 21:30:28.676572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.676548 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4e8889-10b0-4a81-84c0-dc2332cd082a-kube-api-access-jk65m\" (UniqueName: \"kubernetes.io/projected/da4e8889-10b0-4a81-84c0-dc2332cd082a-kube-api-access-jk65m\") pod \"da4e8889-10b0-4a81-84c0-dc2332cd082a\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " Nov 02 21:30:28.676680 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.676594 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4e8889-10b0-4a81-84c0-dc2332cd082a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4e8889-10b0-4a81-84c0-dc2332cd082a-maas-api-service-ca\") pod \"da4e8889-10b0-4a81-84c0-dc2332cd082a\" (UID: \"da4e8889-10b0-4a81-84c0-dc2332cd082a\") " Nov 02 21:30:28.676964 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.676937 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4e8889-10b0-4a81-84c0-dc2332cd082a-maas-api-service-ca" pod "da4e8889-10b0-4a81-84c0-dc2332cd082a" (UID: "da4e8889-10b0-4a81-84c0-dc2332cd082a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:28.678621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.678600 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4e8889-10b0-4a81-84c0-dc2332cd082a-kube-api-access-jk65m" pod "da4e8889-10b0-4a81-84c0-dc2332cd082a" (UID: "da4e8889-10b0-4a81-84c0-dc2332cd082a"). InnerVolumeSpecName "kube-api-access-jk65m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:28.777747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.777673 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4e8889-10b0-4a81-84c0-dc2332cd082a-maas-api-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:28.777747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:28.777707 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk65m\" (UniqueName: \"kubernetes.io/projected/da4e8889-10b0-4a81-84c0-dc2332cd082a-kube-api-access-jk65m\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:29.458130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:29.458103 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" Nov 02 21:30:29.460639 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:29.458132 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9" event={"ID":"da4e8889-10b0-4a81-84c0-dc2332cd082a","Type":"ContainerDied","Data":"8b4b7158837a2257727ac9a20d781e86e7932d4f8742805f6386359fb485c5a5"} Nov 02 21:30:29.460639 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:29.458188 4510 scope.go:122] "RemoveContainer" containerID="88bb076f30418d3fb227cc7e295663c597cca1884a2b223920c842a2bedce348" Nov 02 21:30:29.609020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:29.608990 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9"] Nov 02 21:30:29.611451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:29.611428 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-c2dd3b-29894250-gl6m9"] Nov 02 21:30:30.153512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.153478 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f975752-56ae-4b16-9f14-d122c358139e" path="/var/lib/kubelet/pods/1f975752-56ae-4b16-9f14-d122c358139e/volumes" Nov 02 21:30:30.153932 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.153917 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" path="/var/lib/kubelet/pods/da4e8889-10b0-4a81-84c0-dc2332cd082a/volumes" Nov 02 21:30:30.847047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.847017 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:30:30.946696 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.946659 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps"] Nov 02 21:30:30.947202 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947189 4510 cpu_manager.go:414] "containerMap: removing container" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:30.947245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947203 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:30.947245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947222 4510 cpu_manager.go:414] "containerMap: removing container" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947227 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947236 4510 cpu_manager.go:414] "containerMap: removing container" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947241 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947249 4510 cpu_manager.go:414] "containerMap: removing container" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947254 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947262 4510 cpu_manager.go:414] "containerMap: removing container" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947267 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947286 4510 cpu_manager.go:414] "containerMap: removing container" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947291 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947311 4510 cpu_manager.go:414] "containerMap: removing container" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947316 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947426 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="312e36a7-e1b0-4eb0-9f4e-9ecd047a6e8c" containerName="istio-proxy" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947439 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947447 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947454 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4e8889-10b0-4a81-84c0-dc2332cd082a" containerName="cleanup" Nov 02 21:30:30.947773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947462 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerName="istio-proxy" Nov 02 21:30:30.947773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947468 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:30.947773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947474 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:30.947773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.947480 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:30.952556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:30.951158 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.002729 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.002702 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-socket\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.002894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.002734 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-certs\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.002894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.002798 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-podinfo\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.002894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.002824 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-token\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-token\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.002894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.002881 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-credential-socket\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.003131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003028 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-envoy\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.003131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003076 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79616d36-9d8b-4e32-bcbf-01ec99576118-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79616d36-9d8b-4e32-bcbf-01ec99576118-istiod-ca-cert\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.003131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003108 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-data\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.003293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003179 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-kube-api-access-qbkfb\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-kube-api-access-qbkfb\") pod \"79616d36-9d8b-4e32-bcbf-01ec99576118\" (UID: \"79616d36-9d8b-4e32-bcbf-01ec99576118\") " Nov 02 21:30:31.003293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003237 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4d9ea0-openshift-default-dockercfg-vfpmv\"" Nov 02 21:30:31.003411 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003071 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-certs" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:31.003411 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003097 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-socket" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:31.003892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003866 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-credential-socket" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:31.003997 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003963 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79616d36-9d8b-4e32-bcbf-01ec99576118-istiod-ca-cert" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:30:31.003997 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.003988 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.004128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.004010 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.004128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.004027 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.004128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.004084 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-data" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:31.005498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.005471 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-envoy" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:30:31.005689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.005652 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-podinfo" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:30:31.006069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.006039 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-token" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:31.007702 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.006881 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-kube-api-access-qbkfb" pod "79616d36-9d8b-4e32-bcbf-01ec99576118" (UID: "79616d36-9d8b-4e32-bcbf-01ec99576118"). InnerVolumeSpecName "kube-api-access-qbkfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:30:31.055015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.054987 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps"] Nov 02 21:30:31.105071 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105041 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105098 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105222 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105263 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105351 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105314 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105398 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105380 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105427 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105469 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105451 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-292d4\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-kube-api-access-292d4\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105587 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105572 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25ff849-68d8-495c-8e8a-78016c3c20cd-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.105693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105675 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.105728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105698 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79616d36-9d8b-4e32-bcbf-01ec99576118-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.105728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105708 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.105728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105717 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbkfb\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-kube-api-access-qbkfb\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.105816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105728 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.105816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.105736 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79616d36-9d8b-4e32-bcbf-01ec99576118-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:30:31.206952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.206874 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.206952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.206921 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207060 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207166 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207208 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207288 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207267 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207306 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-292d4\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-kube-api-access-292d4\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207409 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207351 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25ff849-68d8-495c-8e8a-78016c3c20cd-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207409 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207388 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207433 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207510 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207675 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207268 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.207793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.207775 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.208108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.208088 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25ff849-68d8-495c-8e8a-78016c3c20cd-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.209494 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.209465 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.209585 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.209570 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.260258 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.260232 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.260396 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.260379 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-292d4\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-kube-api-access-292d4\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.269877 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.269850 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:31.471682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.471595 4510 generic.go:356] "Generic (PLEG): container finished" podID="79616d36-9d8b-4e32-bcbf-01ec99576118" containerID="694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6" exitCode=0 Nov 02 21:30:31.471682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.471659 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" Nov 02 21:30:31.471882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.471678 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" event={"ID":"79616d36-9d8b-4e32-bcbf-01ec99576118","Type":"ContainerDied","Data":"694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6"} Nov 02 21:30:31.471882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.471733 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9" event={"ID":"79616d36-9d8b-4e32-bcbf-01ec99576118","Type":"ContainerDied","Data":"35c174c3a3ff99ffe5bfc910b40a9acf68f379eee77e8bf8e42b497db11d8075"} Nov 02 21:30:31.471882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.471760 4510 scope.go:122] "RemoveContainer" containerID="694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6" Nov 02 21:30:31.481287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.481270 4510 scope.go:122] "RemoveContainer" containerID="694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6" Nov 02 21:30:31.481582 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:31.481565 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6\": container with ID starting with 694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6 not found: ID does not exist" containerID="694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6" Nov 02 21:30:31.481626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.481591 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6"} err="failed to get container status \"694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6\": rpc error: code = NotFound desc = could not find container \"694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6\": container with ID starting with 694c697e186e415ebeb52cf82447f1c51eb12ba973057f3caea5d6550221d5a6 not found: ID does not exist" Nov 02 21:30:31.625752 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.625721 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9"] Nov 02 21:30:31.629184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.629161 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-j9dd9"] Nov 02 21:30:31.696432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.696401 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps"] Nov 02 21:30:31.703952 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:30:31.703928 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25ff849_68d8_495c_8e8a_78016c3c20cd.slice/crio-999d153a5ac5a062cba82f2750b898e72143c857200289f768435613745cfceb WatchSource:0}: Error finding container 999d153a5ac5a062cba82f2750b898e72143c857200289f768435613745cfceb: Status 404 returned error can't find the container with id 999d153a5ac5a062cba82f2750b898e72143c857200289f768435613745cfceb Nov 02 21:30:31.706544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.706497 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:31.706649 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.706577 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:31.706649 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:31.706609 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:32.153439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:32.153407 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79616d36-9d8b-4e32-bcbf-01ec99576118" path="/var/lib/kubelet/pods/79616d36-9d8b-4e32-bcbf-01ec99576118/volumes" Nov 02 21:30:32.478851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:32.478772 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" event={"ID":"a25ff849-68d8-495c-8e8a-78016c3c20cd","Type":"ContainerStarted","Data":"ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd"} Nov 02 21:30:32.478851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:32.478822 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" event={"ID":"a25ff849-68d8-495c-8e8a-78016c3c20cd","Type":"ContainerStarted","Data":"999d153a5ac5a062cba82f2750b898e72143c857200289f768435613745cfceb"} Nov 02 21:30:32.588499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:32.588444 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podStartSLOduration=2.588430702 podStartE2EDuration="2.588430702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:32.586271495 +0000 UTC m=+1058.562569274" watchObservedRunningTime="2026-11-02 21:30:32.588430702 +0000 UTC m=+1058.564728477" Nov 02 21:30:33.270604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:33.270561 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:33.272499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:33.272473 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:30:33.272635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:33.272542 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:30:34.270462 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:34.270432 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:30:34.270648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:34.270482 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:30:35.270892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:35.270859 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:30:35.271285 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:35.270921 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:30:35.663035 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:35.663005 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:30:36.270953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:36.270916 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:30:36.271340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:36.270981 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:30:37.270556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:37.270510 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:30:37.270730 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:37.270586 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:30:38.270210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:38.270177 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 02 21:30:38.270610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:38.270230 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 02 21:30:39.274461 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:39.274431 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:39.274862 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:39.274833 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:39.275514 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:39.275493 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:30:42.384792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.384760 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85"] Nov 02 21:30:42.385325 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.385311 4510 cpu_manager.go:414] "containerMap: removing container" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:42.385367 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.385327 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f975752-56ae-4b16-9f14-d122c358139e" containerName="cleanup" Nov 02 21:30:42.394424 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.394405 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.442330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.442299 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85"] Nov 02 21:30:42.455995 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.455968 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-home\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.456109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.456014 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-dshm\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.456166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.456131 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e53dc8e1-072d-4586-a65a-9417b18bbe44-tls-certs\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.456166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.456155 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9mj\" (UniqueName: \"kubernetes.io/projected/e53dc8e1-072d-4586-a65a-9417b18bbe44-kube-api-access-js9mj\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.456255 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.456174 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-tmp-dir\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.456255 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.456193 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-model-cache\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.485457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.485429 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"openshift-service-ca.crt\"" Nov 02 21:30:42.485583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.485491 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"default-dockercfg-kqqs4\"" Nov 02 21:30:42.485583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.485514 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"kube-root-ca.crt\"" Nov 02 21:30:42.488576 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.488556 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-284b8e\"/\"distinct-w6-284b8e-kserve-self-signed-certs\"" Nov 02 21:30:42.558178 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558144 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-home\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558198 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-dshm\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558283 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e53dc8e1-072d-4586-a65a-9417b18bbe44-tls-certs\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558311 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9mj\" (UniqueName: \"kubernetes.io/projected/e53dc8e1-072d-4586-a65a-9417b18bbe44-kube-api-access-js9mj\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558337 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-tmp-dir\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558461 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-model-cache\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558673 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558653 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-home\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558742 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-tmp-dir\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.558837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.558814 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-model-cache\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.560470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.560448 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-dshm\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.560764 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.560749 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e53dc8e1-072d-4586-a65a-9417b18bbe44-tls-certs\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.611147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.611122 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9mj\" (UniqueName: \"kubernetes.io/projected/e53dc8e1-072d-4586-a65a-9417b18bbe44-kube-api-access-js9mj\") pod \"distinct-w6-284b8e-kserve-7dc855589b-qpd85\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:42.706867 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:42.706798 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:43.105094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:43.105014 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85"] Nov 02 21:30:43.112403 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:30:43.112373 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode53dc8e1_072d_4586_a65a_9417b18bbe44.slice/crio-1e9099cb6a4b64158d8dcf2a3ec4847e7befe04db0025f2a30964f07726d3403 WatchSource:0}: Error finding container 1e9099cb6a4b64158d8dcf2a3ec4847e7befe04db0025f2a30964f07726d3403: Status 404 returned error can't find the container with id 1e9099cb6a4b64158d8dcf2a3ec4847e7befe04db0025f2a30964f07726d3403 Nov 02 21:30:43.549128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:43.549093 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" event={"ID":"e53dc8e1-072d-4586-a65a-9417b18bbe44","Type":"ContainerStarted","Data":"17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0"} Nov 02 21:30:43.549508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:43.549137 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" event={"ID":"e53dc8e1-072d-4586-a65a-9417b18bbe44","Type":"ContainerStarted","Data":"1e9099cb6a4b64158d8dcf2a3ec4847e7befe04db0025f2a30964f07726d3403"} Nov 02 21:30:43.656385 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:43.656335 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podStartSLOduration=1.656322715 podStartE2EDuration="1.656322715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:43.653668688 +0000 UTC m=+1069.629966476" watchObservedRunningTime="2026-11-02 21:30:43.656322715 +0000 UTC m=+1069.632620498" Nov 02 21:30:43.706909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:43.706872 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:45.533607 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:30:45.533569 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:30:47.220396 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.220365 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm"] Nov 02 21:30:47.224463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.224445 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.315927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.315891 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093783da-0908-4409-a259-ebdc628d6e07-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.315944 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx556\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-kube-api-access-qx556\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.316049 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.316082 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.316139 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.316187 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093783da-0908-4409-a259-ebdc628d6e07-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.316213 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.316304 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.316435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.316328 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.321498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.321472 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm"] Nov 02 21:30:47.418205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418174 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418205 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418267 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093783da-0908-4409-a259-ebdc628d6e07-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418289 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx556\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-kube-api-access-qx556\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418322 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418353 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418407 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418456 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093783da-0908-4409-a259-ebdc628d6e07-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418483 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418814 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418706 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418814 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418723 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.418945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.418925 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.419036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.419016 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.419279 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.419256 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093783da-0908-4409-a259-ebdc628d6e07-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.420966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.420938 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093783da-0908-4409-a259-ebdc628d6e07-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.421121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.421099 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.471016 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.470938 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.471409 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.471392 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx556\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-kube-api-access-qx556\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.537909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.537875 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:47.962567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.962535 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm"] Nov 02 21:30:47.969279 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:30:47.969251 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093783da_0908_4409_a259_ebdc628d6e07.slice/crio-e8517cb8e26ae5317fc2ffcec355b68d527039313a05168f2d354159369e41d8 WatchSource:0}: Error finding container e8517cb8e26ae5317fc2ffcec355b68d527039313a05168f2d354159369e41d8: Status 404 returned error can't find the container with id e8517cb8e26ae5317fc2ffcec355b68d527039313a05168f2d354159369e41d8 Nov 02 21:30:47.971904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.971869 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:47.971985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.971953 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:47.972024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:47.971992 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:30:48.579077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:48.579036 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" event={"ID":"093783da-0908-4409-a259-ebdc628d6e07","Type":"ContainerStarted","Data":"ab5a05dbffabdfec21aba619327af656dd8708c9f031aa115c524925e5c39b34"} Nov 02 21:30:48.579458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:48.579103 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" event={"ID":"093783da-0908-4409-a259-ebdc628d6e07","Type":"ContainerStarted","Data":"e8517cb8e26ae5317fc2ffcec355b68d527039313a05168f2d354159369e41d8"} Nov 02 21:30:48.691625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:48.691574 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podStartSLOduration=1.6915579699999999 podStartE2EDuration="1.69155797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:48.688816849 +0000 UTC m=+1074.665114645" watchObservedRunningTime="2026-11-02 21:30:48.69155797 +0000 UTC m=+1074.667855744" Nov 02 21:30:49.541531 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:49.541479 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:49.541884 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:49.541859 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:30:49.542053 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:49.542023 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:30:50.548661 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:50.548621 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:30:50.549058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:50.548685 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:30:51.541544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:51.539863 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:30:51.541544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:51.539926 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:30:52.176991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.176961 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf"] Nov 02 21:30:52.181339 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.181315 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.190682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.190662 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tls-certs\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.190875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.190854 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-home\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.190986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.190976 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-dshm\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.191148 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.191136 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-model-cache\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.191308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.191292 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tmp-dir\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.191442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.191423 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zc6v\" (UniqueName: \"kubernetes.io/projected/c85f72e4-bab3-4754-8f49-463cc0c4daa9-kube-api-access-9zc6v\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.292700 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.292665 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-home\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.292860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.292707 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-dshm\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.292860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.292760 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-model-cache\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.292860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.292788 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tmp-dir\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.292860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.292804 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zc6v\" (UniqueName: \"kubernetes.io/projected/c85f72e4-bab3-4754-8f49-463cc0c4daa9-kube-api-access-9zc6v\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.292992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.292862 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tls-certs\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.293268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.293250 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-home\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.293765 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.293742 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tmp-dir\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.293884 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.293856 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-model-cache\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.295333 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.295312 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-dshm\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.305557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.305509 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"distinct-w2-4d9ea0-kserve-self-signed-certs\"" Nov 02 21:30:52.306059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.306045 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"openshift-service-ca.crt\"" Nov 02 21:30:52.308685 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.308666 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"kube-root-ca.crt\"" Nov 02 21:30:52.308685 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.308680 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4d9ea0\"/\"default-dockercfg-fpf4h\"" Nov 02 21:30:52.308890 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.308874 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf"] Nov 02 21:30:52.315915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.315890 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tls-certs\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.378742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.378229 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zc6v\" (UniqueName: \"kubernetes.io/projected/c85f72e4-bab3-4754-8f49-463cc0c4daa9-kube-api-access-9zc6v\") pod \"distinct-w2-4d9ea0-kserve-65779867cb-x6jgf\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.495477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.495404 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:52.546543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.545929 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:30:52.546543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.545995 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:30:52.707201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.707171 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:52.719906 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.719880 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:52.952363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:52.952327 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf"] Nov 02 21:30:52.953738 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:30:52.953714 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc85f72e4_bab3_4754_8f49_463cc0c4daa9.slice/crio-7cb397aeab74815a37eee894a677fc696dfbae80ab6e367b8d272eb5189acb2b WatchSource:0}: Error finding container 7cb397aeab74815a37eee894a677fc696dfbae80ab6e367b8d272eb5189acb2b: Status 404 returned error can't find the container with id 7cb397aeab74815a37eee894a677fc696dfbae80ab6e367b8d272eb5189acb2b Nov 02 21:30:53.548994 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:53.548959 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:30:53.549485 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:53.549020 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:30:53.619380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:53.619335 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" event={"ID":"c85f72e4-bab3-4754-8f49-463cc0c4daa9","Type":"ContainerStarted","Data":"3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f"} Nov 02 21:30:53.619380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:53.619376 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" event={"ID":"c85f72e4-bab3-4754-8f49-463cc0c4daa9","Type":"ContainerStarted","Data":"7cb397aeab74815a37eee894a677fc696dfbae80ab6e367b8d272eb5189acb2b"} Nov 02 21:30:53.655293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:53.655267 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:30:53.747544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:53.728323 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podStartSLOduration=1.7282840099999999 podStartE2EDuration="1.72828401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:30:53.726857443 +0000 UTC m=+1079.703155219" watchObservedRunningTime="2026-11-02 21:30:53.72828401 +0000 UTC m=+1079.704581786" Nov 02 21:30:54.496424 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:54.496391 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:30:54.538254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:54.538221 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 21:30:54.538435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:54.538291 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 21:30:55.542819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:55.542788 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:55.543187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:55.543161 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:30:55.544007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:30:55.543991 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:31:01.568416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.568381 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd"] Nov 02 21:31:01.573744 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.573723 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.668617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.668583 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"openshift-service-ca.crt\"" Nov 02 21:31:01.668617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.668606 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"distinct-w1-c2dd3b-kserve-self-signed-certs\"" Nov 02 21:31:01.668897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.668665 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"default-dockercfg-lnkl5\"" Nov 02 21:31:01.668897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.668745 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"kube-root-ca.crt\"" Nov 02 21:31:01.671345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.671326 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd"] Nov 02 21:31:01.709606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.709576 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-tmp-dir\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.709725 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.709624 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9px6b\" (UniqueName: \"kubernetes.io/projected/fe4486fa-40c4-4a43-90b4-07e436859993-kube-api-access-9px6b\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.709725 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.709700 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-model-cache\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.709814 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.709732 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-home\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.709862 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.709849 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-dshm\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.709931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.709917 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4486fa-40c4-4a43-90b4-07e436859993-tls-certs\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811012 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.810976 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-tmp-dir\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811012 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811016 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9px6b\" (UniqueName: \"kubernetes.io/projected/fe4486fa-40c4-4a43-90b4-07e436859993-kube-api-access-9px6b\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811040 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-model-cache\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811077 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-home\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811141 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-dshm\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811216 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4486fa-40c4-4a43-90b4-07e436859993-tls-certs\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811424 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811407 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-tmp-dir\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811473 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811423 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-model-cache\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.811535 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.811498 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-home\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.813413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.813392 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-dshm\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.813594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.813575 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4486fa-40c4-4a43-90b4-07e436859993-tls-certs\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.863695 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.863669 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9px6b\" (UniqueName: \"kubernetes.io/projected/fe4486fa-40c4-4a43-90b4-07e436859993-kube-api-access-9px6b\") pod \"distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:01.885749 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:01.885725 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:02.254582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.254478 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd"] Nov 02 21:31:02.261398 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:31:02.261368 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe4486fa_40c4_4a43_90b4_07e436859993.slice/crio-c9aad43002306cfe6e13d1a924a35db25ea5434fcc34395a45537a65d27468fa WatchSource:0}: Error finding container c9aad43002306cfe6e13d1a924a35db25ea5434fcc34395a45537a65d27468fa: Status 404 returned error can't find the container with id c9aad43002306cfe6e13d1a924a35db25ea5434fcc34395a45537a65d27468fa Nov 02 21:31:02.496323 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.496285 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:31:02.509233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.509156 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:31:02.674184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.674148 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" event={"ID":"fe4486fa-40c4-4a43-90b4-07e436859993","Type":"ContainerStarted","Data":"f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e"} Nov 02 21:31:02.674607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.674198 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" event={"ID":"fe4486fa-40c4-4a43-90b4-07e436859993","Type":"ContainerStarted","Data":"c9aad43002306cfe6e13d1a924a35db25ea5434fcc34395a45537a65d27468fa"} Nov 02 21:31:02.685533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.685499 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:31:02.782072 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.781965 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podStartSLOduration=1.781951609 podStartE2EDuration="1.781951609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:02.779051148 +0000 UTC m=+1088.755348920" watchObservedRunningTime="2026-11-02 21:31:02.781951609 +0000 UTC m=+1088.758249429" Nov 02 21:31:02.886682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:02.886653 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:11.490236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:11.490189 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:31:11.886267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:11.886237 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:11.899253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:11.899227 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:12.770186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:12.770154 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:31:15.529414 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:31:15.529378 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:31:19.856925 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:19.856877 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9"] Nov 02 21:31:19.860978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:19.860955 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:19.958072 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:19.958035 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"openshift-service-ca.crt\"" Nov 02 21:31:19.958252 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:19.958082 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"kube-root-ca.crt\"" Nov 02 21:31:19.958252 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:19.958213 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"default-dockercfg-fgsj4\"" Nov 02 21:31:19.958252 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:19.958240 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ef6b43\"/\"distinct-w5-ef6b43-kserve-self-signed-certs\"" Nov 02 21:31:19.960973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:19.960953 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9"] Nov 02 21:31:20.030936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.030901 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-model-cache\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.031109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.031024 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tmp-dir\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.031109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.031045 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jhlw\" (UniqueName: \"kubernetes.io/projected/552a7878-b45f-4a9d-a82d-90618e9ac7a9-kube-api-access-2jhlw\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.031182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.031131 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-dshm\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.031182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.031155 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tls-certs\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.031292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.031279 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-home\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.131931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.131896 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-home\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.131970 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-model-cache\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.132071 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tmp-dir\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.132097 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jhlw\" (UniqueName: \"kubernetes.io/projected/552a7878-b45f-4a9d-a82d-90618e9ac7a9-kube-api-access-2jhlw\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.132170 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-dshm\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.132200 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tls-certs\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132388 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.132351 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-home\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.132390 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-model-cache\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.132551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.132455 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tmp-dir\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.134436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.134414 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-dshm\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.134869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.134847 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tls-certs\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.184855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.184827 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jhlw\" (UniqueName: \"kubernetes.io/projected/552a7878-b45f-4a9d-a82d-90618e9ac7a9-kube-api-access-2jhlw\") pod \"distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.193724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.193702 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:20.595392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.595302 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9"] Nov 02 21:31:20.602230 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:31:20.602202 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod552a7878_b45f_4a9d_a82d_90618e9ac7a9.slice/crio-7fa4b25ed05cd06ad1e99a83c551a83b6947657627e0413160ac619c2ea75b73 WatchSource:0}: Error finding container 7fa4b25ed05cd06ad1e99a83c551a83b6947657627e0413160ac619c2ea75b73: Status 404 returned error can't find the container with id 7fa4b25ed05cd06ad1e99a83c551a83b6947657627e0413160ac619c2ea75b73 Nov 02 21:31:20.785590 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.785556 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" event={"ID":"552a7878-b45f-4a9d-a82d-90618e9ac7a9","Type":"ContainerStarted","Data":"da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0"} Nov 02 21:31:20.785793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.785601 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" event={"ID":"552a7878-b45f-4a9d-a82d-90618e9ac7a9","Type":"ContainerStarted","Data":"7fa4b25ed05cd06ad1e99a83c551a83b6947657627e0413160ac619c2ea75b73"} Nov 02 21:31:20.787948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.787921 4510 generic.go:356] "Generic (PLEG): container finished" podID="77756a42-43cc-4d6f-a2a7-8c889058c214" containerID="3068f6422d9e0c4b4abaf2ab5be16fe20e69b4c99cc20988dc41e73b8785f853" exitCode=1 Nov 02 21:31:20.788066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.787952 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" event={"ID":"77756a42-43cc-4d6f-a2a7-8c889058c214","Type":"ContainerDied","Data":"3068f6422d9e0c4b4abaf2ab5be16fe20e69b4c99cc20988dc41e73b8785f853"} Nov 02 21:31:20.788249 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.788235 4510 scope.go:122] "RemoveContainer" containerID="3068f6422d9e0c4b4abaf2ab5be16fe20e69b4c99cc20988dc41e73b8785f853" Nov 02 21:31:20.895642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:20.895598 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podStartSLOduration=1.895583315 podStartE2EDuration="1.895583315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:20.89484903 +0000 UTC m=+1106.871146809" watchObservedRunningTime="2026-11-02 21:31:20.895583315 +0000 UTC m=+1106.871881090" Nov 02 21:31:21.194041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:21.194008 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:21.797024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:21.796984 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-jvrm4" event={"ID":"77756a42-43cc-4d6f-a2a7-8c889058c214","Type":"ContainerStarted","Data":"b35cac4de1d50ede6de845bf0c8aee2bd605f6b1fc333e240e05b3119fc84806"} Nov 02 21:31:30.194077 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:30.194045 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:30.206616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:30.206592 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:30.858482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:30.858450 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:31:43.250005 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.249928 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f"] Nov 02 21:31:43.254180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.254161 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.350205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.350177 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"openshift-service-ca.crt\"" Nov 02 21:31:43.350387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.350372 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"default-dockercfg-ptsrn\"" Nov 02 21:31:43.350457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.350402 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"distinct-2-w3-9a6cf2-kserve-self-signed-certs\"" Nov 02 21:31:43.350683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.350666 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9a6cf2\"/\"kube-root-ca.crt\"" Nov 02 21:31:43.353406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.353380 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f"] Nov 02 21:31:43.405619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.405509 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-home\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.406047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.406019 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-tmp-dir\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.406192 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.406096 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-dshm\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.406253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.406236 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-model-cache\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.406314 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.406268 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baea62ff-7e44-4e92-83b4-018d05ecc226-tls-certs\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.407967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.406322 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bd2j\" (UniqueName: \"kubernetes.io/projected/baea62ff-7e44-4e92-83b4-018d05ecc226-kube-api-access-9bd2j\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507351 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-dshm\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507453 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-model-cache\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507475 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baea62ff-7e44-4e92-83b4-018d05ecc226-tls-certs\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507505 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bd2j\" (UniqueName: \"kubernetes.io/projected/baea62ff-7e44-4e92-83b4-018d05ecc226-kube-api-access-9bd2j\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507594 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-home\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507617 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-tmp-dir\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507929 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507908 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-model-cache\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507951 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-home\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.507987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.507963 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-tmp-dir\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.509633 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.509605 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-dshm\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.509907 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.509887 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baea62ff-7e44-4e92-83b4-018d05ecc226-tls-certs\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.560194 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.560168 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bd2j\" (UniqueName: \"kubernetes.io/projected/baea62ff-7e44-4e92-83b4-018d05ecc226-kube-api-access-9bd2j\") pod \"distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.565612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.565592 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:43.958705 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:43.958671 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f"] Nov 02 21:31:43.968136 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:31:43.968104 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaea62ff_7e44_4e92_83b4_018d05ecc226.slice/crio-8c8f5027e76fe0d60f0032929e17d978d0de6e424be5a85f4ef7ee2482372662 WatchSource:0}: Error finding container 8c8f5027e76fe0d60f0032929e17d978d0de6e424be5a85f4ef7ee2482372662: Status 404 returned error can't find the container with id 8c8f5027e76fe0d60f0032929e17d978d0de6e424be5a85f4ef7ee2482372662 Nov 02 21:31:44.944596 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:44.944555 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" event={"ID":"baea62ff-7e44-4e92-83b4-018d05ecc226","Type":"ContainerStarted","Data":"78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42"} Nov 02 21:31:44.945009 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:44.944606 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" event={"ID":"baea62ff-7e44-4e92-83b4-018d05ecc226","Type":"ContainerStarted","Data":"8c8f5027e76fe0d60f0032929e17d978d0de6e424be5a85f4ef7ee2482372662"} Nov 02 21:31:45.054416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:45.054351 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podStartSLOduration=2.054333439 podStartE2EDuration="2.054333439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:45.049798364 +0000 UTC m=+1131.026096140" watchObservedRunningTime="2026-11-02 21:31:45.054333439 +0000 UTC m=+1131.030631215" Nov 02 21:31:45.534665 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:31:45.534624 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:31:45.566190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:45.566157 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:46.471905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.471875 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb"] Nov 02 21:31:46.476741 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.476721 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.550900 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.550863 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551062 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.550920 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.551101 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551169 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.551129 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.551172 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.551240 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.551362 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.551410 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclks\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-kube-api-access-sclks\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.551555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.551535 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e897ad-5975-4ab4-a508-a574208a7aef-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.560840 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.560816 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb"] Nov 02 21:31:46.652626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.652594 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.652813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.652705 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.652813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.652724 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.652813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.652760 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.652813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.652782 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.652825 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.652953 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclks\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-kube-api-access-sclks\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.653012 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e897ad-5975-4ab4-a508-a574208a7aef-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653204 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.653060 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653204 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.653178 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.653231 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.653292 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.653404 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.653739 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.653720 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e897ad-5975-4ab4-a508-a574208a7aef-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.655079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.655051 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.655249 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.655228 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.706782 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.706755 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.706986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.706966 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclks\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-kube-api-access-sclks\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:46.790968 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:46.790897 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:47.218281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:47.218249 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb"] Nov 02 21:31:47.225220 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:31:47.225192 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e897ad_5975_4ab4_a508_a574208a7aef.slice/crio-9a587e165c4f77acf7d0398a712f14ec3e3e077f554d8914cec1f4845d2179db WatchSource:0}: Error finding container 9a587e165c4f77acf7d0398a712f14ec3e3e077f554d8914cec1f4845d2179db: Status 404 returned error can't find the container with id 9a587e165c4f77acf7d0398a712f14ec3e3e077f554d8914cec1f4845d2179db Nov 02 21:31:47.227570 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:47.227508 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:47.227645 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:47.227604 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:47.227645 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:47.227634 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:31:47.963929 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:47.963833 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" event={"ID":"f5e897ad-5975-4ab4-a508-a574208a7aef","Type":"ContainerStarted","Data":"cea251b13d9b9e00890eba1a649f5c352af2848d6f29cad5c6efa8740d9540f5"} Nov 02 21:31:47.963929 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:47.963900 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" event={"ID":"f5e897ad-5975-4ab4-a508-a574208a7aef","Type":"ContainerStarted","Data":"9a587e165c4f77acf7d0398a712f14ec3e3e077f554d8914cec1f4845d2179db"} Nov 02 21:31:48.165032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:48.164983 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podStartSLOduration=2.164969189 podStartE2EDuration="2.164969189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:31:48.116954677 +0000 UTC m=+1134.093252451" watchObservedRunningTime="2026-11-02 21:31:48.164969189 +0000 UTC m=+1134.141266964" Nov 02 21:31:48.569293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:48.569256 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:31:48.791224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:48.791194 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:48.792578 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:48.792551 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:31:48.792725 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:48.792604 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 21:31:49.791978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:49.791949 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:31:49.792373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:49.791996 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 21:31:50.791727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:50.791692 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:31:50.792017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:50.791744 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 21:31:51.791792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:51.791761 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:31:51.791959 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:51.791811 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 21:31:52.792107 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:52.792069 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:31:52.792491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:52.792127 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 21:31:53.566292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:53.566252 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:53.578951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:53.578917 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:53.791234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:53.791196 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 21:31:53.791407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:53.791263 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 21:31:54.018245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:54.018217 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:31:54.796577 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:54.796550 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:54.796819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:54.796793 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:31:54.797581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:31:54.797563 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:32:15.531557 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:32:15.531491 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:32:18.430315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:18.430282 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:32:31.519788 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.519752 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq"] Nov 02 21:32:31.525085 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.525064 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.605783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.605756 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.605951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.605889 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.605951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.605930 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.606029 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.605957 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxchb\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-kube-api-access-cxchb\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.606029 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.605977 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.606094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.606050 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.606128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.606094 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.606190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.606166 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.606251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.606236 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.620291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.620262 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq"] Nov 02 21:32:31.706858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.706828 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.706868 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.706913 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.706930 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707032 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707063 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707126 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxchb\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-kube-api-access-cxchb\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707323 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707293 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707418 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707472 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707561 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707599 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.707711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.707643 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.708054 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.708034 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.709579 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.709557 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.709787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.709772 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.760420 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.760395 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.760733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.760710 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxchb\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-kube-api-access-cxchb\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:31.838399 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:31.838334 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:32.271742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:32.271712 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq"] Nov 02 21:32:32.278589 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:32:32.278562 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4947166_47c6_4322_a3d5_c9e4ec483dc6.slice/crio-026adc65d67cf700a090e6a6ad8bbaba047561af0a20f7f835c4c41d886849f9 WatchSource:0}: Error finding container 026adc65d67cf700a090e6a6ad8bbaba047561af0a20f7f835c4c41d886849f9: Status 404 returned error can't find the container with id 026adc65d67cf700a090e6a6ad8bbaba047561af0a20f7f835c4c41d886849f9 Nov 02 21:32:32.280801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:32.280773 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:32.280870 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:32.280840 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:32.280908 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:32.280887 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:33.020379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.020347 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n"] Nov 02 21:32:33.029567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.024652 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.082616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.082570 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n"] Nov 02 21:32:33.149382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149327 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.149623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149464 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.149623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149554 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.149755 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149682 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.149821 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149752 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.149821 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149790 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3764eb-eac8-4784-8e81-f935b19bd206-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.149933 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149886 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.149933 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149923 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxckp\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-kube-api-access-hxckp\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.150033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.149961 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.252627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252482 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.252627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252611 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.252881 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252693 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.252881 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252768 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.252881 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252803 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3764eb-eac8-4784-8e81-f935b19bd206-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.252989 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252945 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.253024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252990 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxckp\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-kube-api-access-hxckp\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.253057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.253032 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.253126 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.253100 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.253572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.252951 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.253704 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.253588 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.253777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.253715 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.254671 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.254649 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3764eb-eac8-4784-8e81-f935b19bd206-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.254790 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.254747 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.255799 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.255774 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.256953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.256929 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.258668 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.258644 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" event={"ID":"d4947166-47c6-4322-a3d5-c9e4ec483dc6","Type":"ContainerStarted","Data":"e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6"} Nov 02 21:32:33.258768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.258692 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" event={"ID":"d4947166-47c6-4322-a3d5-c9e4ec483dc6","Type":"ContainerStarted","Data":"026adc65d67cf700a090e6a6ad8bbaba047561af0a20f7f835c4c41d886849f9"} Nov 02 21:32:33.306818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.306760 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxckp\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-kube-api-access-hxckp\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.307212 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.307191 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.357147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.357113 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:33.372944 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.372887 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podStartSLOduration=2.3728693229999998 podStartE2EDuration="2.372869323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:33.36607881 +0000 UTC m=+1179.342376590" watchObservedRunningTime="2026-11-02 21:32:33.372869323 +0000 UTC m=+1179.349167099" Nov 02 21:32:33.792454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.792419 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n"] Nov 02 21:32:33.805994 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:32:33.805961 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3764eb_eac8_4784_8e81_f935b19bd206.slice/crio-33eface3593910b93740c9afa2410da4231e95631b84e94e523ccf7718fd53a5 WatchSource:0}: Error finding container 33eface3593910b93740c9afa2410da4231e95631b84e94e523ccf7718fd53a5: Status 404 returned error can't find the container with id 33eface3593910b93740c9afa2410da4231e95631b84e94e523ccf7718fd53a5 Nov 02 21:32:33.808606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.808574 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:33.808695 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.808639 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:33.808695 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.808667 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:32:33.850746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.846900 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:33.852036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.851928 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:32:33.852036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:33.851980 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:32:34.266915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.266872 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" event={"ID":"ed3764eb-eac8-4784-8e81-f935b19bd206","Type":"ContainerStarted","Data":"bdca68259a992fb38b93c31052a9f898a6f9b8b149c2b5a7c29c22a31738dddc"} Nov 02 21:32:34.267341 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.266938 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" event={"ID":"ed3764eb-eac8-4784-8e81-f935b19bd206","Type":"ContainerStarted","Data":"33eface3593910b93740c9afa2410da4231e95631b84e94e523ccf7718fd53a5"} Nov 02 21:32:34.357944 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.357911 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:34.359591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.359563 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:32:34.359746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.359612 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:32:34.379834 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.379774 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podStartSLOduration=2.379756169 podStartE2EDuration="2.379756169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:32:34.376445148 +0000 UTC m=+1180.352742922" watchObservedRunningTime="2026-11-02 21:32:34.379756169 +0000 UTC m=+1180.356053942" Nov 02 21:32:34.840558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.840028 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:32:34.840558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:34.840094 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:32:35.358543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:35.358089 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:32:35.358543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:35.358154 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:32:35.840909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:35.840871 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:32:35.841181 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:35.841154 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:32:36.363546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:36.361952 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:32:36.363546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:36.362015 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:32:36.839024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:36.838910 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:32:36.839298 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:36.839253 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:32:37.360542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:37.359901 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:32:37.360542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:37.359972 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:32:37.848543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:37.848065 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:32:37.848543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:37.848127 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:32:38.357479 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:38.357439 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:32:38.357683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:38.357496 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:32:38.839544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:38.838817 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 02 21:32:38.839544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:38.838880 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 02 21:32:39.357989 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:39.357956 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:32:39.358485 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:39.358008 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:32:39.862480 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:39.862448 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:39.862926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:39.862897 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:39.863855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:39.863836 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:32:40.358294 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:40.358206 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 02 21:32:40.358294 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:40.358259 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 02 21:32:41.362797 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:41.362769 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:41.363291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:41.363117 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:41.364256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:41.364236 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:32:45.532027 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:32:45.531988 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:32:48.690859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:32:48.690825 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:33:15.530957 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:33:15.530881 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:33:23.220308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:23.220278 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:33:25.326543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:25.324637 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:33:25.326543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:25.324735 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:25.856543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:25.855990 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 21:33:26.349992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.349921 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv"] Nov 02 21:33:26.365072 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.364559 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.409880 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.409796 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv"] Nov 02 21:33:26.450272 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.450238 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c2dd3b\"/\"distinct-2-w1-c2dd3b-kserve-self-signed-certs\"" Nov 02 21:33:26.516200 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.516152 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-home\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.516398 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.516245 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-tmp-dir\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.516398 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.516381 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-dshm\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.516560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.516479 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77hg\" (UniqueName: \"kubernetes.io/projected/2a38d552-966f-4ca0-8f60-be2db4e772fe-kube-api-access-v77hg\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.516560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.516510 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a38d552-966f-4ca0-8f60-be2db4e772fe-tls-certs\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.516728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.516595 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-model-cache\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618119 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-dshm\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618296 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618178 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77hg\" (UniqueName: \"kubernetes.io/projected/2a38d552-966f-4ca0-8f60-be2db4e772fe-kube-api-access-v77hg\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618296 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618198 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a38d552-966f-4ca0-8f60-be2db4e772fe-tls-certs\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618296 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618234 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-model-cache\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618296 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618268 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-home\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618329 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-tmp-dir\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618698 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-model-cache\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618730 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-home\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.618829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.618767 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-tmp-dir\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.620454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.620433 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-dshm\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.620847 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.620829 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a38d552-966f-4ca0-8f60-be2db4e772fe-tls-certs\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.671073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.671046 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77hg\" (UniqueName: \"kubernetes.io/projected/2a38d552-966f-4ca0-8f60-be2db4e772fe-kube-api-access-v77hg\") pod \"distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:26.707599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:26.707577 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:27.112735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:27.112710 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv"] Nov 02 21:33:27.119488 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:33:27.119452 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a38d552_966f_4ca0_8f60_be2db4e772fe.slice/crio-c109385a461c5a0c4df74329d94a2b8a5395ebfc0340e8c4bdffc9adcd86a4cd WatchSource:0}: Error finding container c109385a461c5a0c4df74329d94a2b8a5395ebfc0340e8c4bdffc9adcd86a4cd: Status 404 returned error can't find the container with id c109385a461c5a0c4df74329d94a2b8a5395ebfc0340e8c4bdffc9adcd86a4cd Nov 02 21:33:27.670023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:27.669991 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" event={"ID":"2a38d552-966f-4ca0-8f60-be2db4e772fe","Type":"ContainerStarted","Data":"60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab"} Nov 02 21:33:27.670511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:27.670040 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" event={"ID":"2a38d552-966f-4ca0-8f60-be2db4e772fe","Type":"ContainerStarted","Data":"c109385a461c5a0c4df74329d94a2b8a5395ebfc0340e8c4bdffc9adcd86a4cd"} Nov 02 21:33:27.707998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:27.707970 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:27.776432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:27.776381 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podStartSLOduration=1.776369802 podStartE2EDuration="1.776369802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:27.7756838 +0000 UTC m=+1233.751981585" watchObservedRunningTime="2026-11-02 21:33:27.776369802 +0000 UTC m=+1233.752667577" Nov 02 21:33:31.982688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:31.982653 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh"] Nov 02 21:33:31.986943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:31.986924 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.086033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.085988 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh"] Nov 02 21:33:32.091343 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091314 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.091504 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091462 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.091613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091596 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.091685 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091631 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.091759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091745 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.091820 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091809 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5j9b\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-kube-api-access-z5j9b\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.091936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091912 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.091994 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.091977 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.092037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.092026 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193022 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.192991 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193194 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193041 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193194 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193152 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193225 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193285 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193357 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193343 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193417 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193362 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193417 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193392 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193428 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193502 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5j9b\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-kube-api-access-z5j9b\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193505 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-certs\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193636 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-data\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.193728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.193667 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-credential-socket\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.194173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.194155 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istiod-ca-cert\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.195623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.195602 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-envoy\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.195861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.195842 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-podinfo\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.245383 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.245324 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-token\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.245512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.245492 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5j9b\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-kube-api-access-z5j9b\") pod \"e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.301356 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.301324 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:32.728477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.728448 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh"] Nov 02 21:33:32.735221 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:33:32.735198 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e318998_65bd_40f8_be06_9e02a4a6b8aa.slice/crio-86696cb682780abb341788be0e76d5865eb4171b76994ca464402af4c70f6b46 WatchSource:0}: Error finding container 86696cb682780abb341788be0e76d5865eb4171b76994ca464402af4c70f6b46: Status 404 returned error can't find the container with id 86696cb682780abb341788be0e76d5865eb4171b76994ca464402af4c70f6b46 Nov 02 21:33:32.737729 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.737696 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:32.737786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.737759 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:32.737829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:32.737816 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:33.711380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.711336 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" event={"ID":"2e318998-65bd-40f8-be06-9e02a4a6b8aa","Type":"ContainerStarted","Data":"e9f558c80f1156d3ae31a7f21d8c94861e3e87788760f6a33205761b2fbf0d51"} Nov 02 21:33:33.711832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.711401 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" event={"ID":"2e318998-65bd-40f8-be06-9e02a4a6b8aa","Type":"ContainerStarted","Data":"86696cb682780abb341788be0e76d5865eb4171b76994ca464402af4c70f6b46"} Nov 02 21:33:33.866985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.866921 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podStartSLOduration=2.866904482 podStartE2EDuration="2.866904482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:33.865447631 +0000 UTC m=+1239.841745447" watchObservedRunningTime="2026-11-02 21:33:33.866904482 +0000 UTC m=+1239.843202258" Nov 02 21:33:33.915858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.915826 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj"] Nov 02 21:33:33.920154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.920131 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:33.955275 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.955247 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj"] Nov 02 21:33:33.968919 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.968855 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"kube-root-ca.crt\"" Nov 02 21:33:33.968919 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.968855 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"openshift-service-ca.crt\"" Nov 02 21:33:33.969088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.968916 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"distinct-2-w4-4e2b27-kserve-self-signed-certs\"" Nov 02 21:33:33.969088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:33.968951 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e2b27\"/\"default-dockercfg-44bw2\"" Nov 02 21:33:34.021048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.019035 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tls-certs\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.021048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.019215 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-model-cache\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.021048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.019366 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-dshm\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.021048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.019443 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tmp-dir\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.021048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.019511 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clsf8\" (UniqueName: \"kubernetes.io/projected/7f6d7a08-3da9-4f23-80a6-faba18eb394d-kube-api-access-clsf8\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.021048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.019658 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-home\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121312 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tmp-dir\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121356 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clsf8\" (UniqueName: \"kubernetes.io/projected/7f6d7a08-3da9-4f23-80a6-faba18eb394d-kube-api-access-clsf8\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121429 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-home\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121478 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tls-certs\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121758 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121602 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-model-cache\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121758 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121700 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-dshm\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121870 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121818 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tmp-dir\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.121870 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121851 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-home\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.122042 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.121954 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-model-cache\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.142542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.136751 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-dshm\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.142542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.140115 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tls-certs\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.177013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.176982 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clsf8\" (UniqueName: \"kubernetes.io/projected/7f6d7a08-3da9-4f23-80a6-faba18eb394d-kube-api-access-clsf8\") pod \"distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.265683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.262595 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:34.308579 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.304023 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:34.308579 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.304406 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:33:34.308579 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.304473 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:33:34.704205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.704169 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj"] Nov 02 21:33:34.710794 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:33:34.710758 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f6d7a08_3da9_4f23_80a6_faba18eb394d.slice/crio-c51e2c261d4f749167dcf554c3db0f4772fcc625e13a2dda4563cecbf122f1ec WatchSource:0}: Error finding container c51e2c261d4f749167dcf554c3db0f4772fcc625e13a2dda4563cecbf122f1ec: Status 404 returned error can't find the container with id c51e2c261d4f749167dcf554c3db0f4772fcc625e13a2dda4563cecbf122f1ec Nov 02 21:33:34.719965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:34.719940 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" event={"ID":"7f6d7a08-3da9-4f23-80a6-faba18eb394d","Type":"ContainerStarted","Data":"c51e2c261d4f749167dcf554c3db0f4772fcc625e13a2dda4563cecbf122f1ec"} Nov 02 21:33:35.302343 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:35.302306 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:33:35.302553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:35.302365 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:33:35.748552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:35.745158 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" event={"ID":"7f6d7a08-3da9-4f23-80a6-faba18eb394d","Type":"ContainerStarted","Data":"85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6"} Nov 02 21:33:35.853085 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:35.853032 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podStartSLOduration=2.853018792 podStartE2EDuration="2.853018792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:35.850922136 +0000 UTC m=+1241.827219914" watchObservedRunningTime="2026-11-02 21:33:35.853018792 +0000 UTC m=+1241.829316566" Nov 02 21:33:36.263330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:36.263289 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:36.301951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:36.301922 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:33:36.302083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:36.301971 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:33:36.708407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:36.708373 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:36.721544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:36.721503 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:36.768767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:36.768734 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:37.302376 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:37.302345 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:33:37.302571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:37.302395 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:33:38.302014 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:38.301979 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:33:38.302394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:38.302028 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:33:39.302344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:39.302309 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 21:33:39.302760 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:39.302361 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 21:33:40.306055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:40.306025 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:40.306555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:40.306515 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:40.307139 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:40.307121 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:33:44.263129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:44.263088 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:44.279080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:44.279053 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:44.836079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:44.836051 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:45.533163 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:33:45.533134 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:33:48.577428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.577392 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz"] Nov 02 21:33:48.581984 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.581960 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738246 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738212 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738294 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738339 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738357 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb869\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-kube-api-access-jb869\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738377 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738415 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738611 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738645 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.738787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.738748 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.790904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.790862 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz"] Nov 02 21:33:48.839950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.839864 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.839950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.839905 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.839950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.839938 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.839955 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb869\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-kube-api-access-jb869\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.839975 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.839997 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840104 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840135 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840233 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840306 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-credential-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840410 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-certs\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840666 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-data\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840713 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-socket\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.840989 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.840967 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istiod-ca-cert\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.842210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.842189 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-envoy\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.842470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.842453 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-podinfo\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.892443 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.892417 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-token\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.892704 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.892680 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb869\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-kube-api-access-jb869\") pod \"e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:48.895432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:48.895402 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:49.324044 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:49.324013 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz"] Nov 02 21:33:49.330439 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:33:49.330413 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5271f4_91ba_4ddf_b35f_a1b06f503d32.slice/crio-ff308a8a435293223c969fa375b97d46efcefecdd511ecd92982362c371804f6 WatchSource:0}: Error finding container ff308a8a435293223c969fa375b97d46efcefecdd511ecd92982362c371804f6: Status 404 returned error can't find the container with id ff308a8a435293223c969fa375b97d46efcefecdd511ecd92982362c371804f6 Nov 02 21:33:49.333005 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:49.332974 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:49.333076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:49.333039 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:49.333076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:49.333067 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:33:49.858601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:49.858560 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" event={"ID":"ee5271f4-91ba-4ddf-b35f-a1b06f503d32","Type":"ContainerStarted","Data":"59e5503ef564c727fab6a49705700280993e8b0458d4de733a0677dabb0a4abc"} Nov 02 21:33:49.859016 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:49.858659 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" event={"ID":"ee5271f4-91ba-4ddf-b35f-a1b06f503d32","Type":"ContainerStarted","Data":"ff308a8a435293223c969fa375b97d46efcefecdd511ecd92982362c371804f6"} Nov 02 21:33:49.895758 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:49.895726 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:50.013936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:50.013882 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podStartSLOduration=2.013868312 podStartE2EDuration="2.013868312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:33:50.012900098 +0000 UTC m=+1255.989197873" watchObservedRunningTime="2026-11-02 21:33:50.013868312 +0000 UTC m=+1255.990166087" Nov 02 21:33:50.898203 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:50.898168 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:33:50.898639 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:50.898234 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:33:51.896272 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:51.896242 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:33:51.896446 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:51.896292 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:33:52.896302 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:52.896273 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:33:52.896703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:52.896321 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:33:53.896713 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:53.896681 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:33:53.897094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:53.896731 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:33:54.896363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:54.896324 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:33:54.896578 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:54.896380 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:33:55.899675 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:55.899634 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:33:55.900291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:55.900260 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:33:56.895940 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:56.895904 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 02 21:33:56.896208 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:56.896183 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 02 21:33:57.643659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:57.643609 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.720665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:57.720622 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.725614 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:57.725580 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.755630 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:57.755590 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.764281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:57.764240 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.833041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:57.832999 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:57.858643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:57.858599 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 21:33:58.007060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.006959 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.883544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.820843 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:58.883544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.843645 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.883544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.843993 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 21:33:58.883544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.844241 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 21:33:58.883544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.876195 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.885846 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.885939 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.885973 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.886006 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.886039 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.895395 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.908703 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.912835 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:33:58.943095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.914056 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:58.967540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.960371 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:33:58.967853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:58.967831 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:33:59.017631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:59.001723 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:33:59.017631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:59.009751 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:33:59.040540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:59.040501 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:33:59.098181 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:59.098152 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:33:59.127605 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:33:59.127510 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:34:02.507288 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.507247 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5"] Nov 02 21:34:02.513117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.513098 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.586996 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587235 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587045 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587320 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587274 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87732d55-1a01-4d5d-b699-19edbff59b35-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587375 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587333 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587419 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587470 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kr45\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-kube-api-access-5kr45\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587584 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587509 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87732d55-1a01-4d5d-b699-19edbff59b35-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587640 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587594 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.587759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.587740 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.688937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.688899 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689018 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689042 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689164 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87732d55-1a01-4d5d-b699-19edbff59b35-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689207 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689336 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689277 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689336 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689317 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kr45\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-kube-api-access-5kr45\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689448 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689355 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87732d55-1a01-4d5d-b699-19edbff59b35-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689448 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689394 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689575 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689503 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689575 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689540 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-certs\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689610 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-credential-socket\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.689777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.689760 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-data\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.690121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.690104 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87732d55-1a01-4d5d-b699-19edbff59b35-istiod-ca-cert\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.691452 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.691431 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-envoy\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.691600 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.691584 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87732d55-1a01-4d5d-b699-19edbff59b35-istio-podinfo\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.702687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.702663 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5"] Nov 02 21:34:02.742424 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.742113 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-istio-token\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.742424 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.742353 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kr45\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-kube-api-access-5kr45\") pod \"e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:02.827023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:02.826935 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:03.260071 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:03.260033 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5"] Nov 02 21:34:03.265173 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:34:03.265149 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87732d55_1a01_4d5d_b699_19edbff59b35.slice/crio-b3b2a48ffbed287e96faaf72f84b18e1c69a0c1b3eee0db4d432452d90c9c031 WatchSource:0}: Error finding container b3b2a48ffbed287e96faaf72f84b18e1c69a0c1b3eee0db4d432452d90c9c031: Status 404 returned error can't find the container with id b3b2a48ffbed287e96faaf72f84b18e1c69a0c1b3eee0db4d432452d90c9c031 Nov 02 21:34:03.267800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:03.267765 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:03.267908 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:03.267836 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:03.267908 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:03.267864 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:04.094229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:04.094191 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" event={"ID":"87732d55-1a01-4d5d-b699-19edbff59b35","Type":"ContainerStarted","Data":"ca02590464fa2805068da9bd7f82a7179031f1d754f3a0e1fc91efbf26a2ab90"} Nov 02 21:34:04.094709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:04.094242 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" event={"ID":"87732d55-1a01-4d5d-b699-19edbff59b35","Type":"ContainerStarted","Data":"b3b2a48ffbed287e96faaf72f84b18e1c69a0c1b3eee0db4d432452d90c9c031"} Nov 02 21:34:04.254577 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:04.254497 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podStartSLOduration=2.254477655 podStartE2EDuration="2.254477655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:04.250876788 +0000 UTC m=+1270.227174564" watchObservedRunningTime="2026-11-02 21:34:04.254477655 +0000 UTC m=+1270.230775431" Nov 02 21:34:04.848455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:04.848419 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:04.848893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:04.848865 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:34:04.849142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:04.849096 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:34:05.844551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:05.838811 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:34:05.844551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:05.838881 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:34:06.845351 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:06.844878 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:34:06.845351 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:06.844949 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:34:07.840910 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:07.840875 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:34:07.841194 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:07.841162 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:34:08.831544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:08.827811 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:34:08.831544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:08.827867 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:34:09.827804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:09.827770 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 21:34:09.827976 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:09.827822 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 21:34:10.831917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:10.831888 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:10.832311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:10.832259 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:10.833015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:10.832993 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:34:15.530866 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:34:15.530826 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:34:17.044495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.044452 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m"] Nov 02 21:34:17.049388 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.049364 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103193 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103293 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103389 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103365 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103386 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103572 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bnzg\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-kube-api-access-4bnzg\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103616 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103642 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103764 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.103837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.103810 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.174112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.174086 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m"] Nov 02 21:34:17.205291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205263 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bnzg\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-kube-api-access-4bnzg\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205304 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205324 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205374 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205407 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205474 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205696 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205573 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205696 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205674 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205699 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.205953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.205930 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.206229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.206205 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.206405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.206384 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.206568 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.206471 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.206716 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.206699 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.208386 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.208362 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.208469 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.208429 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.262215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.262181 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.269356 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.269335 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bnzg\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-kube-api-access-4bnzg\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.363738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.363708 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:17.794302 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.794224 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m"] Nov 02 21:34:17.804731 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.804694 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:17.804816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.804774 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:17.804853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.804815 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:17.883871 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.883841 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb"] Nov 02 21:34:17.889380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.889359 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:17.941507 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:17.941476 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb"] Nov 02 21:34:18.016865 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.016833 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017045 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.016887 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017045 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.016906 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017045 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.016974 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017045 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.017021 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b866e72-25c8-4171-ad22-f5eff84395b7-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.017116 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.017150 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdf6\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-kube-api-access-srdf6\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.017251 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.017481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.017333 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118051 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118096 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdf6\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-kube-api-access-srdf6\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118166 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118257 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118368 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118412 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118436 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118460 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118532 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118492 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b866e72-25c8-4171-ad22-f5eff84395b7-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118970 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118639 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118970 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118809 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118970 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118843 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.118970 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.118964 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.119299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.119278 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b866e72-25c8-4171-ad22-f5eff84395b7-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.120578 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.120558 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.120765 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.120744 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.170947 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.170921 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.171084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.170985 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdf6\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-kube-api-access-srdf6\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.195830 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.195794 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" event={"ID":"c78ab5d8-f481-4ac1-9d2a-5553b735f385","Type":"ContainerStarted","Data":"f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16"} Nov 02 21:34:18.195970 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.195856 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" event={"ID":"c78ab5d8-f481-4ac1-9d2a-5553b735f385","Type":"ContainerStarted","Data":"f8b019338cb0ad6475f1b55c1149aeeefdeeb9c496a6c77b7e25281a41c1ae33"} Nov 02 21:34:18.205534 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.205500 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:18.216130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.216106 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn"] Nov 02 21:34:18.221858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.221837 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.234907 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.234882 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr"] Nov 02 21:34:18.242236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.242203 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.277491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.277466 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn"] Nov 02 21:34:18.298571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.296335 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr"] Nov 02 21:34:18.310565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.308550 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podStartSLOduration=1.308536184 podStartE2EDuration="1.308536184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:18.306487087 +0000 UTC m=+1284.282784866" watchObservedRunningTime="2026-11-02 21:34:18.308536184 +0000 UTC m=+1284.284833956" Nov 02 21:34:18.321334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321060 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.321334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321146 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.321334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321170 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fec9655-c924-4228-808e-78533a4c500d-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.321334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321289 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.321334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321331 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321399 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvxh\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-kube-api-access-nmvxh\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321440 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321472 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321512 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321568 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fec9655-c924-4228-808e-78533a4c500d-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321593 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321666 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.321727 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321710 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.322136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321762 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.322136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321793 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.322136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321842 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdzd\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-kube-api-access-7kdzd\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.322136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321876 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.322136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.321901 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.377606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.371587 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:18.377606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.377022 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:18.377606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.377094 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:18.423504 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423474 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.423756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423512 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.423756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423616 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvxh\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-kube-api-access-nmvxh\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.423756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423654 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.423756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423687 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.423756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423720 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.424025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423759 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fec9655-c924-4228-808e-78533a4c500d-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.424025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423788 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.424025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423865 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.424025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423911 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.424025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.423969 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.424025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424002 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.424334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424041 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdzd\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-kube-api-access-7kdzd\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.424334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424086 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.424334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424126 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.424334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424243 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.424334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424303 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.424334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424304 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.424334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424328 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fec9655-c924-4228-808e-78533a4c500d-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.425548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.424815 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.425548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.425365 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.427379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.426116 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.427379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.426363 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.427379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.427085 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.428310 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.427960 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.428310 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.428223 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.429060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.429039 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fec9655-c924-4228-808e-78533a4c500d-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.429779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.429746 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.430803 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.430746 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.433186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.433139 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.433767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.433746 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.455496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.454678 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fec9655-c924-4228-808e-78533a4c500d-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.477654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.477621 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.477872 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.477733 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvxh\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-kube-api-access-nmvxh\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.479115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.479088 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdzd\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-kube-api-access-7kdzd\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.480250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.480229 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.565000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.564967 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:18.571455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.571429 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:18.597168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.597142 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb"] Nov 02 21:34:18.602320 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:34:18.602284 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b866e72_25c8_4171_ad22_f5eff84395b7.slice/crio-163075d63d64601883c4f26621300b858b3e0de417bf5de0ea7e05c8ed522313 WatchSource:0}: Error finding container 163075d63d64601883c4f26621300b858b3e0de417bf5de0ea7e05c8ed522313: Status 404 returned error can't find the container with id 163075d63d64601883c4f26621300b858b3e0de417bf5de0ea7e05c8ed522313 Nov 02 21:34:18.605155 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.605110 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.605233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.605206 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.605273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.605249 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.965411 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.965306 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn"] Nov 02 21:34:18.972902 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:34:18.972873 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6617d0_5ecb_4db5_bc87_d7d9f5611af3.slice/crio-8652ec1e7a8705fd867973c34801be2d41809dc5a801dc4fb6c1147036908c3a WatchSource:0}: Error finding container 8652ec1e7a8705fd867973c34801be2d41809dc5a801dc4fb6c1147036908c3a: Status 404 returned error can't find the container with id 8652ec1e7a8705fd867973c34801be2d41809dc5a801dc4fb6c1147036908c3a Nov 02 21:34:18.976646 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.976595 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.976989 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.976696 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.976989 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.976736 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.987573 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.987549 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr"] Nov 02 21:34:18.994206 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:34:18.994186 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fec9655_c924_4228_808e_78533a4c500d.slice/crio-410805569b5cbb12102c4e84c106f400dfff209b4e6a94be1894b9dfc65664e3 WatchSource:0}: Error finding container 410805569b5cbb12102c4e84c106f400dfff209b4e6a94be1894b9dfc65664e3: Status 404 returned error can't find the container with id 410805569b5cbb12102c4e84c106f400dfff209b4e6a94be1894b9dfc65664e3 Nov 02 21:34:18.997161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.997129 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.997238 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.997202 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:18.997281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:18.997236 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:19.213268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.213224 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" event={"ID":"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3","Type":"ContainerStarted","Data":"121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d"} Nov 02 21:34:19.213781 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.213290 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" event={"ID":"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3","Type":"ContainerStarted","Data":"8652ec1e7a8705fd867973c34801be2d41809dc5a801dc4fb6c1147036908c3a"} Nov 02 21:34:19.216537 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.216449 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" event={"ID":"9fec9655-c924-4228-808e-78533a4c500d","Type":"ContainerStarted","Data":"b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c"} Nov 02 21:34:19.216718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.216701 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" event={"ID":"9fec9655-c924-4228-808e-78533a4c500d","Type":"ContainerStarted","Data":"410805569b5cbb12102c4e84c106f400dfff209b4e6a94be1894b9dfc65664e3"} Nov 02 21:34:19.221630 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.221603 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" event={"ID":"5b866e72-25c8-4171-ad22-f5eff84395b7","Type":"ContainerStarted","Data":"c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54"} Nov 02 21:34:19.221757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.221653 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" event={"ID":"5b866e72-25c8-4171-ad22-f5eff84395b7","Type":"ContainerStarted","Data":"163075d63d64601883c4f26621300b858b3e0de417bf5de0ea7e05c8ed522313"} Nov 02 21:34:19.363513 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.363445 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podStartSLOduration=1.363422731 podStartE2EDuration="1.363422731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:19.344829227 +0000 UTC m=+1285.321127013" watchObservedRunningTime="2026-11-02 21:34:19.363422731 +0000 UTC m=+1285.339720510" Nov 02 21:34:19.378544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.365964 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:19.378544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.377198 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:19.456541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.456413 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podStartSLOduration=1.456390354 podStartE2EDuration="1.456390354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:19.453998352 +0000 UTC m=+1285.430296127" watchObservedRunningTime="2026-11-02 21:34:19.456390354 +0000 UTC m=+1285.432688131" Nov 02 21:34:19.570601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.569601 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:19.577138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.577064 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:19.579371 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:19.579309 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podStartSLOduration=2.579289689 podStartE2EDuration="2.579289689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:19.564007914 +0000 UTC m=+1285.540305726" watchObservedRunningTime="2026-11-02 21:34:19.579289689 +0000 UTC m=+1285.555587465" Nov 02 21:34:20.215539 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.212920 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:20.219538 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.217889 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:34:20.219538 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.217962 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 21:34:20.367545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.364647 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:20.367545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.364719 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:20.576763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.576661 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:34:20.577067 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.577020 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 21:34:20.587666 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.587634 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:34:20.587951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.587905 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:34:20.985544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:20.974667 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:21.008545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.007605 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:21.030544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.029310 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:21.052542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.049664 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:21.105544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.104693 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:34:21.130606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.129600 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:21.206545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.200601 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:21.210539 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.208918 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:34:21.210539 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.208971 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 21:34:21.386545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.371711 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:21.386545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.371783 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:21.572542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.571975 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:34:21.572542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.572061 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:34:21.572542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.572405 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:34:21.572542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.572436 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 21:34:21.983544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.979971 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:21.983544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:21.980742 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:34:22.007543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.006603 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:22.026542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.008594 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:22.026542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.008967 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:34:22.029586 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.029553 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:22.030811 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.029951 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:34:22.061663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.061619 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 21:34:22.062308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.062285 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:34:22.105492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.105451 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:22.106140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.106117 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:34:22.137545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.132591 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:22.137545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.133070 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:34:22.379604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.338327 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:34:22.379604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.338392 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 21:34:22.455279 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.392721 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:22.455279 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.392777 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:22.766599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.749073 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:34:22.766599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.749134 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:34:22.766599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.749214 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:34:22.766599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.749237 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 21:34:22.936379 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.914513 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:22.989322 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:22.989277 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.018613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.014674 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:34:23.018613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.015126 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:34:23.018613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.015956 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.044593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.044555 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.066992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.062789 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.106910 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.106870 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.141783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.141722 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.299331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.212686 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:34:23.359664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.359627 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:34:23.359929 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.359907 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 21:34:23.368544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.367780 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:23.368544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.367832 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:23.510653 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.510593 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.654582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.639605 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:34:23.654582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.639677 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 21:34:23.730886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.730852 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:34:23.823551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.798236 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:23.919058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.918972 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:34:23.919297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.919267 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:34:24.008610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:23.992716 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 21:34:24.042690 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.042639 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:24.611972 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.611912 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:24.653567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.650324 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:24.653567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.650386 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:24.674295 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.656070 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:24.678548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.674544 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 21:34:24.782382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.724817 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:34:24.782382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.724872 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 21:34:24.782382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.724945 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:34:24.782382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.724968 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:34:24.857751 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.857718 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:34:24.857999 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:24.857970 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 21:34:25.000451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.000407 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:25.222950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.222911 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:34:25.223228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.223197 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 21:34:25.301687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.301604 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:34:25.318349 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.318313 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:25.452549 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.442193 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 21:34:25.452894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.452861 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 21:34:25.584599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.568925 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:34:25.584599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.569000 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 21:34:25.584599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.573965 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:34:25.584599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.574021 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:34:25.595340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.595301 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 21:34:25.634547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.622603 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 21:34:25.658181 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.647227 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:25.695551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.680886 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 21:34:25.695551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.681270 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 21:34:25.695551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.694900 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:34:25.940073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.940047 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:34:25.968609 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:25.968577 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:34:26.038437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.038408 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:34:26.213705 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.206220 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 21:34:26.213705 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.206277 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 21:34:26.553001 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.552539 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:26.553299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.553245 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:26.554427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.554407 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:34:26.566274 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.565994 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 21:34:26.566274 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.566048 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 21:34:26.572429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.572401 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 21:34:26.572594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:26.572455 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 21:34:27.322540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.321861 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:27.326537 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.323029 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:27.326537 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.324872 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:34:27.727273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.727243 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:27.727601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.727578 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:27.729022 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.728997 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:34:27.783427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.783388 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:27.784130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.784103 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:27.785020 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:27.784996 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:34:32.998209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:32.998173 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx"] Nov 02 21:34:33.004946 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.004916 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.062301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.062270 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx"] Nov 02 21:34:33.180877 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.180841 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5hr\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-kube-api-access-sm5hr\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181052 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.180975 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.181050 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.181106 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.181157 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181263 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.181242 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181377 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.181359 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.181448 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.181545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.181512 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.282926 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283049 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5hr\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-kube-api-access-sm5hr\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283117 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283155 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283221 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283273 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283350 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283402 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283561 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283541 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-certs\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283494 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283706 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.283936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.283909 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-credential-socket\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.284080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.284061 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-data\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.284205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.284180 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istiod-ca-cert\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.285697 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.285671 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-envoy\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.285893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.285876 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-podinfo\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.338481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.338441 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5hr\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-kube-api-access-sm5hr\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.339425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.339402 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-token\") pod \"e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:33.623506 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:33.623480 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:34.051217 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.051127 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx"] Nov 02 21:34:34.057587 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:34:34.057556 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a339a5_da47_4fdc_9cf5_7d5ba07b735d.slice/crio-cb3d5f329409b912d773b1e7d8d01437ae3ee39f5a6cf2a2e4aa1d17a3b459bf WatchSource:0}: Error finding container cb3d5f329409b912d773b1e7d8d01437ae3ee39f5a6cf2a2e4aa1d17a3b459bf: Status 404 returned error can't find the container with id cb3d5f329409b912d773b1e7d8d01437ae3ee39f5a6cf2a2e4aa1d17a3b459bf Nov 02 21:34:34.060160 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.060130 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:34.060234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.060198 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:34.060234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.060228 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:34.778198 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.778157 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" event={"ID":"17a339a5-da47-4fdc-9cf5-7d5ba07b735d","Type":"ContainerStarted","Data":"822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2"} Nov 02 21:34:34.778386 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.778217 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" event={"ID":"17a339a5-da47-4fdc-9cf5-7d5ba07b735d","Type":"ContainerStarted","Data":"cb3d5f329409b912d773b1e7d8d01437ae3ee39f5a6cf2a2e4aa1d17a3b459bf"} Nov 02 21:34:34.953610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.953546 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podStartSLOduration=2.953513241 podStartE2EDuration="2.953513241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:34.948029262 +0000 UTC m=+1300.924327047" watchObservedRunningTime="2026-11-02 21:34:34.953513241 +0000 UTC m=+1300.929811017" Nov 02 21:34:34.998009 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:34.997980 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:34:35.634305 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:35.634107 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:35.634928 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:35.634906 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:34:35.635081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:35.635061 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 21:34:36.629964 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:36.629930 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:34:36.630256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:36.630231 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 21:34:37.628545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:37.627932 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:34:37.628545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:37.627991 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 21:34:37.705100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:37.704621 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 21:34:37.712559 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:37.708391 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:37.729099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:37.728637 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.201912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.137566 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.201912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.137994 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 21:34:38.201912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.138300 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:34:38.225691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.225640 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:34:38.659543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.655656 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:34:38.689000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.688956 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 21:34:38.753683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.744373 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 21:34:38.754266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:38.754239 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:34:39.009605 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:39.006872 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:34:39.637694 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:39.637657 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:34:39.637980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:39.637952 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 21:34:40.644168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:40.644134 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:34:40.645027 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:40.644989 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 21:34:41.624831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:41.624794 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 21:34:41.625025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:41.624895 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 21:34:42.629273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:42.629243 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:42.629737 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:42.629717 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:42.630422 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:42.630402 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:34:45.532449 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:34:45.532406 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:34:48.716123 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.716085 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj"] Nov 02 21:34:48.721362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.721340 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806238 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806205 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk942\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-kube-api-access-vk942\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806256 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806347 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806386 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806411 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806510 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806674 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806710 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.806752 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.806728 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.822286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.822259 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj"] Nov 02 21:34:48.908188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908155 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908215 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908264 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908468 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908482 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908618 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908581 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908618 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908606 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908678 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk942\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-kube-api-access-vk942\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908735 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908844 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908786 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.908844 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908795 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.909001 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.908977 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.909085 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.909021 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.909159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.909141 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.910783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.910758 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.911032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.911015 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.960801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.960774 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:48.961140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:48.961121 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk942\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-kube-api-access-vk942\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:49.035687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:49.035621 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:49.466673 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:49.466638 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj"] Nov 02 21:34:49.473182 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:34:49.473153 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d7eb5a_704b_46c5_91e1_3a50149d06cd.slice/crio-7481f56846c150a7e417eb08d67644ff874628fe90982fb16ee6b3b6a716af93 WatchSource:0}: Error finding container 7481f56846c150a7e417eb08d67644ff874628fe90982fb16ee6b3b6a716af93: Status 404 returned error can't find the container with id 7481f56846c150a7e417eb08d67644ff874628fe90982fb16ee6b3b6a716af93 Nov 02 21:34:49.475908 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:49.475875 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:49.475998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:49.475943 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:49.475998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:49.475975 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:34:49.986230 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:49.986189 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" event={"ID":"08d7eb5a-704b-46c5-91e1-3a50149d06cd","Type":"ContainerStarted","Data":"7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d"} Nov 02 21:34:49.986802 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:49.986254 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" event={"ID":"08d7eb5a-704b-46c5-91e1-3a50149d06cd","Type":"ContainerStarted","Data":"7481f56846c150a7e417eb08d67644ff874628fe90982fb16ee6b3b6a716af93"} Nov 02 21:34:50.043541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:50.042873 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:50.045544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:50.044829 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:34:50.045544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:50.044883 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:34:50.146544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:50.145599 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podStartSLOduration=2.145578508 podStartE2EDuration="2.145578508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:34:50.142757954 +0000 UTC m=+1316.119055743" watchObservedRunningTime="2026-11-02 21:34:50.145578508 +0000 UTC m=+1316.121876289" Nov 02 21:34:51.046541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:51.039845 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:34:51.046541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:51.039901 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:34:52.042544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:52.039859 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:34:52.042544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:52.039919 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:34:53.042540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:53.041857 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:34:53.042540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:53.041957 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:34:54.048982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:54.048926 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:34:54.049510 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:54.048993 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:34:55.049061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:55.047146 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:34:55.049061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:55.047219 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:34:56.036352 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:56.036315 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 21:34:56.036564 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:56.036392 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 21:34:57.040637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:57.040607 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:57.041015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:57.040988 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:34:57.041805 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:34:57.041787 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:35:02.377555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.377491 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj"] Nov 02 21:35:02.384479 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.384454 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.436330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.436291 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj"] Nov 02 21:35:02.534441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.534395 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882bn\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-kube-api-access-882bn\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.534659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.534502 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.534728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.534683 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.534795 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.534728 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.534883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.534860 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.534947 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.534917 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec368-d117-40a2-adf6-ae868a46b083-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.535003 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.534958 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.535113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.535092 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec368-d117-40a2-adf6-ae868a46b083-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.535178 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.535139 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636099 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636143 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec368-d117-40a2-adf6-ae868a46b083-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636258 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636339 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec368-d117-40a2-adf6-ae868a46b083-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636365 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636441 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882bn\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-kube-api-access-882bn\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636486 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636614 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636510 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-data\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636614 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636597 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-certs\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636660 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.636723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636714 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.637010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636986 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.637094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.636993 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec368-d117-40a2-adf6-ae868a46b083-istiod-ca-cert\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.637094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.637051 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-credential-socket\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.638752 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.638726 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-envoy\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.639087 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.639065 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec368-d117-40a2-adf6-ae868a46b083-istio-podinfo\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.689456 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.689424 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-istio-token\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.689620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.689489 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-882bn\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-kube-api-access-882bn\") pod \"e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:02.700756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:02.700734 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:03.132440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:03.132407 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj"] Nov 02 21:35:03.138988 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:35:03.138961 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eec368_d117_40a2_adf6_ae868a46b083.slice/crio-2ab6b98ff576fe1edd7a32cc6fa2356a02e4d50225b77c063f689db2d603593b WatchSource:0}: Error finding container 2ab6b98ff576fe1edd7a32cc6fa2356a02e4d50225b77c063f689db2d603593b: Status 404 returned error can't find the container with id 2ab6b98ff576fe1edd7a32cc6fa2356a02e4d50225b77c063f689db2d603593b Nov 02 21:35:03.141444 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:03.141407 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:03.141565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:03.141490 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:03.141565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:03.141550 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:04.086605 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:04.086570 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" event={"ID":"f8eec368-d117-40a2-adf6-ae868a46b083","Type":"ContainerStarted","Data":"1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75"} Nov 02 21:35:04.087018 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:04.086623 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" event={"ID":"f8eec368-d117-40a2-adf6-ae868a46b083","Type":"ContainerStarted","Data":"2ab6b98ff576fe1edd7a32cc6fa2356a02e4d50225b77c063f689db2d603593b"} Nov 02 21:35:04.296185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:04.296135 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podStartSLOduration=2.2961212460000002 podStartE2EDuration="2.296121246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:04.294684756 +0000 UTC m=+1330.270982531" watchObservedRunningTime="2026-11-02 21:35:04.296121246 +0000 UTC m=+1330.272419020" Nov 02 21:35:04.701107 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:04.701068 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:04.702729 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:04.702697 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:35:04.702881 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:04.702745 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:35:05.701274 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:05.701238 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:35:05.701687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:05.701318 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:35:06.702101 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:06.702057 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:35:06.702587 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:06.702127 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:35:07.701652 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:07.701616 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:35:07.701880 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:07.701677 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:35:08.701738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:08.701703 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:35:08.702139 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:08.701757 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:35:09.701090 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:09.701055 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 02 21:35:09.701260 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:09.701114 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 02 21:35:10.706343 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:10.706309 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:10.706839 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:10.706785 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:10.707536 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:10.707496 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:35:14.055129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:14.055078 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.165946 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:14.126940 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:14.586934 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:14.586893 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Nov 02 21:35:14.707541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:14.706871 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:14.988762 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:14.988723 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:35:14.989076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:14.989041 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.038741 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.037910 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.128842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.104600 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.128842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.105946 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:15.128842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.107065 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:35:15.172916 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.172878 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:15.198634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.173556 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.198634 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.173929 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:35:15.308337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.308075 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Nov 02 21:35:15.382742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.382687 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:15.609257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.609168 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb"] Nov 02 21:35:15.819544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:15.741666 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.054214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.053664 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.142937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.142883 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:16.171659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.154165 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 21:35:16.197777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.197732 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:16.200263 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.198696 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:16.264509 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.264477 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:16.385082 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:35:16.385041 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:35:16.394849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.394818 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2ccb9fb8-gw-openshift-default-dockercfg-cm48w\"" Nov 02 21:35:16.426547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.395279 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:35:16.429417 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.429383 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb"] Nov 02 21:35:16.429715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.429696 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:35:16.429892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.429877 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:35:16.430068 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.430055 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:35:16.504816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.495996 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 21:35:16.505221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.505194 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:35:16.864360 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.826817 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:16.983450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.983404 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-podinfo\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:16.997551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:16.987112 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/888afd07-c4f2-4772-89bc-f6a9f3408041-istiod-ca-cert\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.010549 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.001702 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-socket\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.052586 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.050683 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-token\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.081563 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.057231 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.090664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.090626 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zlb\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-kube-api-access-82zlb\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.109661 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.106992 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-data\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.242594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.183388 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.242594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.183420 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:17.242594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.238411 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-credential-socket\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.242594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.238533 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-envoy\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.242594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.238565 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-certs\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.356454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.356412 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:17.357030 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.356998 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506177 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zlb\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-kube-api-access-82zlb\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506270 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-data\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506366 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-credential-socket\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506432 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-envoy\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506457 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-certs\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506505 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-podinfo\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506577 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/888afd07-c4f2-4772-89bc-f6a9f3408041-istiod-ca-cert\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506636 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-socket\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.517601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.506670 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-token\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.557542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.549062 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-data\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.557542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.549379 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-credential-socket\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.575901 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.575864 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/888afd07-c4f2-4772-89bc-f6a9f3408041-istiod-ca-cert\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.576372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.576349 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-certs\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.577273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.577248 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-socket\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.577460 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.577436 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 02 21:35:17.793598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.773632 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Nov 02 21:35:17.793598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.774716 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zlb\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-kube-api-access-82zlb\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:17.885573 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.867796 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:17.963546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:17.947895 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-envoy\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:18.133611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.082911 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-token\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:18.133611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.091817 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-podinfo\") pod \"e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:18.133611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.107885 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 21:35:18.346543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.310884 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.366545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.316189 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.366957 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.366914 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.385599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.367582 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:18.406950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.375902 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:18.594563 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.589010 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:18.895846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:18.895806 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.133017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.132975 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 02 21:35:19.351203 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.351166 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:19.388117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.373621 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.389336 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.374948 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.416644 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.413095 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:19.619054 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.619012 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:19.621331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.621307 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:35:19.743439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.743385 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:35:19.745215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.745192 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:35:19.752783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.751878 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:35:19.753133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.753108 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:35:19.763096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.755030 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:35:19.763096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:19.757904 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 21:35:20.026092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.026004 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb"] Nov 02 21:35:20.032532 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:35:20.032492 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888afd07_c4f2_4772_89bc_f6a9f3408041.slice/crio-d8f991cb62dae6550fa5c4b088fe275fa4cf9ee766eec4010a949470927ac341 WatchSource:0}: Error finding container d8f991cb62dae6550fa5c4b088fe275fa4cf9ee766eec4010a949470927ac341: Status 404 returned error can't find the container with id d8f991cb62dae6550fa5c4b088fe275fa4cf9ee766eec4010a949470927ac341 Nov 02 21:35:20.035342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.035284 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:20.035489 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.035380 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:20.035489 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.035430 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:20.353754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.353665 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" event={"ID":"888afd07-c4f2-4772-89bc-f6a9f3408041","Type":"ContainerStarted","Data":"0a73413e0f0e8d7a3c421b8ecd52f68867b5f0133cec5c985b8c0efb3c2c687c"} Nov 02 21:35:20.353754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.353724 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" event={"ID":"888afd07-c4f2-4772-89bc-f6a9f3408041","Type":"ContainerStarted","Data":"d8f991cb62dae6550fa5c4b088fe275fa4cf9ee766eec4010a949470927ac341"} Nov 02 21:35:20.384686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.384654 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:20.507839 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:20.507779 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podStartSLOduration=6.507765174 podStartE2EDuration="6.507765174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:20.50263474 +0000 UTC m=+1346.478932514" watchObservedRunningTime="2026-11-02 21:35:20.507765174 +0000 UTC m=+1346.484062948" Nov 02 21:35:21.389845 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:21.389806 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:22.368578 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:22.368549 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:22.369493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:22.369470 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:35:33.770213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.770174 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc"] Nov 02 21:35:33.774545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.774507 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818271 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818247 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818408 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818281 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d22544-4180-49fa-8819-27bd0e34cf6e-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818408 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818379 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpdn\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-kube-api-access-hlpdn\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818429 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818580 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818693 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818757 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818786 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.818810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.818807 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.873756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.873724 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc"] Nov 02 21:35:33.920893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.920863 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.920930 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.920976 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.921010 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.921082 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.921110 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d22544-4180-49fa-8819-27bd0e34cf6e-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.921182 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpdn\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-kube-api-access-hlpdn\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.921228 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921448 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.921337 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-data\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.921448 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.921380 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.922118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.922095 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d22544-4180-49fa-8819-27bd0e34cf6e-istiod-ca-cert\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.922330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.922305 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-certs\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.922400 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.922369 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-credential-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.922644 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.922626 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-socket\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.924304 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.924283 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-podinfo\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.924655 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.924632 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-envoy\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.978347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.978315 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-token\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:33.978490 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:33.978412 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpdn\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-kube-api-access-hlpdn\") pod \"e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:34.073337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.073253 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp"] Nov 02 21:35:34.079272 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.079245 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.096076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.096040 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:34.096624 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.096344 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7"] Nov 02 21:35:34.108258 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.108230 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.254282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.250711 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp"] Nov 02 21:35:34.271991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.271937 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8f8047-52a3-408d-85f1-554d42416e76-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.272287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.272259 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.272451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.272433 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.272597 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.272581 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.272732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.272714 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7jvn\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-kube-api-access-q7jvn\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.272857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.272841 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.272981 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.272968 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8f8047-52a3-408d-85f1-554d42416e76-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.273158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.273141 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.273357 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.273340 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqg6w\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-kube-api-access-fqg6w\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.273477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.273462 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.273615 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.273600 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.273753 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.273739 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.273901 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.273886 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.274055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.274041 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.274206 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.274192 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.274332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.274318 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.274467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.274448 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.274699 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.274681 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.309594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.308364 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7"] Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.375769 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.375825 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqg6w\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-kube-api-access-fqg6w\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.375851 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.375871 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.375900 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.375948 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376003 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376058 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376088 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376121 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376160 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376204 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8f8047-52a3-408d-85f1-554d42416e76-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376238 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376280 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376314 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376320 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376348 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7jvn\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-kube-api-access-q7jvn\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376380 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376394 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376419 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8f8047-52a3-408d-85f1-554d42416e76-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.376619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.376495 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.377582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.377004 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.377582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.377022 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.377582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.377506 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.377948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.377805 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-certs\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.377948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.377914 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-data\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.378114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.378031 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-credential-socket\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.378406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.378383 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8f8047-52a3-408d-85f1-554d42416e76-istiod-ca-cert\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.381010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.380975 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.381098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.381005 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-envoy\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.382570 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.382505 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.383250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.383228 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8f8047-52a3-408d-85f1-554d42416e76-istio-podinfo\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.447404 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.446450 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7jvn\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-kube-api-access-q7jvn\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.448427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.448364 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqg6w\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-kube-api-access-fqg6w\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.453899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.453869 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.454293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.454266 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-token\") pod \"e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.546052 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.546021 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:34.579873 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.579840 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:34.711796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.711763 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc"] Nov 02 21:35:34.719865 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:35:34.719836 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d22544_4180_49fa_8819_27bd0e34cf6e.slice/crio-69f79451e606f7bdf578c706d0de5879f5559fc60bd5cb65f74361f283d269bf WatchSource:0}: Error finding container 69f79451e606f7bdf578c706d0de5879f5559fc60bd5cb65f74361f283d269bf: Status 404 returned error can't find the container with id 69f79451e606f7bdf578c706d0de5879f5559fc60bd5cb65f74361f283d269bf Nov 02 21:35:34.724138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.724092 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:34.724242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.724177 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:34.724242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.724229 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:34.931351 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.931305 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp"] Nov 02 21:35:34.941351 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:35:34.941321 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875f34a7_f57b_4c3b_a15b_a6a2df0f816e.slice/crio-760a1ae7fdca12d195668d987b2a9c40e580ad12f3a8b429e4a5f3449bab717a WatchSource:0}: Error finding container 760a1ae7fdca12d195668d987b2a9c40e580ad12f3a8b429e4a5f3449bab717a: Status 404 returned error can't find the container with id 760a1ae7fdca12d195668d987b2a9c40e580ad12f3a8b429e4a5f3449bab717a Nov 02 21:35:34.944354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.944320 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:34.944482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.944383 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:34.944482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.944415 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:34.988533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:34.988482 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7"] Nov 02 21:35:35.040451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.040413 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:35.040608 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.040509 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:35.040608 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.040568 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:35.562128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.562057 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" event={"ID":"56d22544-4180-49fa-8819-27bd0e34cf6e","Type":"ContainerStarted","Data":"615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70"} Nov 02 21:35:35.562128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.562118 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" event={"ID":"56d22544-4180-49fa-8819-27bd0e34cf6e","Type":"ContainerStarted","Data":"69f79451e606f7bdf578c706d0de5879f5559fc60bd5cb65f74361f283d269bf"} Nov 02 21:35:35.564080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.564055 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" event={"ID":"9a8f8047-52a3-408d-85f1-554d42416e76","Type":"ContainerStarted","Data":"08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9"} Nov 02 21:35:35.564363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.564336 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" event={"ID":"9a8f8047-52a3-408d-85f1-554d42416e76","Type":"ContainerStarted","Data":"e980438e625c460317ce69a1ce29b0eba445c877240d9166c432ac0a6fe060fd"} Nov 02 21:35:35.565711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.565692 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" event={"ID":"875f34a7-f57b-4c3b-a15b-a6a2df0f816e","Type":"ContainerStarted","Data":"f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad"} Nov 02 21:35:35.565762 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.565727 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" event={"ID":"875f34a7-f57b-4c3b-a15b-a6a2df0f816e","Type":"ContainerStarted","Data":"760a1ae7fdca12d195668d987b2a9c40e580ad12f3a8b429e4a5f3449bab717a"} Nov 02 21:35:35.580418 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.580383 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:35.688790 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.688733 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podStartSLOduration=2.68871547 podStartE2EDuration="2.68871547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:35.674195001 +0000 UTC m=+1361.650492805" watchObservedRunningTime="2026-11-02 21:35:35.68871547 +0000 UTC m=+1361.665013245" Nov 02 21:35:35.814028 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.812820 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podStartSLOduration=1.8128010620000001 podStartE2EDuration="1.812801062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:35.807287833 +0000 UTC m=+1361.783585609" watchObservedRunningTime="2026-11-02 21:35:35.812801062 +0000 UTC m=+1361.789098843" Nov 02 21:35:35.954544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:35.947686 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podStartSLOduration=1.947668873 podStartE2EDuration="1.947668873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:35.945902419 +0000 UTC m=+1361.922200198" watchObservedRunningTime="2026-11-02 21:35:35.947668873 +0000 UTC m=+1361.923966649" Nov 02 21:35:36.101611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.097376 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:36.114104 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.114067 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:35:36.114306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.114150 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 21:35:36.559716 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.559676 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 21:35:36.559973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.559945 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 21:35:36.616543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.607160 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p"] Nov 02 21:35:36.616543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.612171 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:35:36.616543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.612311 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:35:36.616543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.614746 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p"] Nov 02 21:35:36.616543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.614793 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:36.616543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.614890 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.697543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.696580 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 21:35:36.697543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.696691 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 21:35:36.697543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.696768 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 21:35:36.697543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.696867 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nwm6f\"" Nov 02 21:35:36.763115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.763070 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.763390 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.763368 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.763595 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.763577 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ba9c8d-b0ea-437a-95fd-b4ce24677947-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.763728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.763710 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.763964 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.763947 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.764123 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.764099 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbrg\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-kube-api-access-ctbrg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.764255 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.764239 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.878686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.876231 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.878686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.876280 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.878686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.876391 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ba9c8d-b0ea-437a-95fd-b4ce24677947-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.878686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.876429 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.878686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.876685 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.878686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.876746 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbrg\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-kube-api-access-ctbrg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.878686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.876802 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.942548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.926785 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:36.993548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.989996 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:37.009541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:36.994341 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ba9c8d-b0ea-437a-95fd-b4ce24677947-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:37.018557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.016946 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:37.018557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.017271 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:37.115540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.112032 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:35:37.115540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.112093 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 21:35:37.130587 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.127093 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:37.158546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.131211 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbrg\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-kube-api-access-ctbrg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bxq4p\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:37.215546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.210507 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:37.416543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.412672 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:37.660602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.564366 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 21:35:37.660602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.564430 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 21:35:37.660602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.592679 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:35:37.660602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.592749 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:35:37.794026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:37.793901 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:38.360674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.360569 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:38.427546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.423430 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:35:38.427546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.423550 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 21:35:38.428223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.428096 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:38.546620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.520774 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:38.587680 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.585402 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:35:38.587680 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.585465 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:35:38.587680 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.587383 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 21:35:38.587680 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.587431 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 21:35:38.811557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.806622 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:38.915132 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.915023 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:38.916457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:38.916424 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:39.049122 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.043030 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:35:39.114092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.109951 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p"] Nov 02 21:35:39.114092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.110076 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:35:39.114092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.110114 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:35:39.119274 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.119252 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 21:35:39.290715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.290628 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:35:39.290987 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.119677 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:35:39.291185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.291149 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 21:35:39.613363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.606762 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 21:35:39.613363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.606842 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 21:35:39.623213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.617558 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:35:39.623213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:39.617601 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:35:40.107598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:40.107204 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:35:40.107598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:40.107262 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 21:35:40.379544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:40.373835 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" event={"ID":"23ba9c8d-b0ea-437a-95fd-b4ce24677947","Type":"ContainerStarted","Data":"9f52515ac394f70efba91f34c25a61e84aff26058aca4b398f79d4a9262eb5a6"} Nov 02 21:35:40.555544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:40.555358 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 21:35:40.555544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:40.555417 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 21:35:40.583546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:40.582869 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:35:40.583546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:40.582927 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:35:41.111488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:41.107187 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:35:41.111488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:41.107245 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 21:35:41.556543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:41.555960 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 21:35:41.556543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:41.556024 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 21:35:41.593544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:41.593134 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:35:41.593544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:41.593202 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:35:42.097624 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:42.097582 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 21:35:42.097841 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:42.097656 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 21:35:42.546696 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:42.546659 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 21:35:42.547070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:42.546726 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 21:35:42.580776 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:42.580617 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:35:42.580776 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:42.580700 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:35:43.661121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.661083 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:43.661810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.661759 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:43.663057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.663036 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:35:43.786048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.785990 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:43.786902 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.786878 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:43.790061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.790020 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:35:43.862567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.858344 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:43.862567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.859626 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:43.862567 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:43.860630 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:35:44.906355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:44.906322 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:44.906672 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:44.906387 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:35:45.532944 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:35:45.532901 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:35:45.571011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:45.570963 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" event={"ID":"23ba9c8d-b0ea-437a-95fd-b4ce24677947","Type":"ContainerStarted","Data":"8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be"} Nov 02 21:35:45.571223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:45.571066 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:45.573583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:45.573556 4510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bxq4p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:45.573702 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:45.573607 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:45.712665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:45.712599 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" podStartSLOduration=3.906046102 podStartE2EDuration="9.71258494s" totalImagesPullingTime="5.806538838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:35:45.71179536 +0000 UTC m=+1371.688093139" watchObservedRunningTime="2026-11-02 21:35:45.71258494 +0000 UTC m=+1371.688882718" Nov 02 21:35:46.629587 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:46.629512 4510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bxq4p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:35:46.630090 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:46.629611 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:35:49.629952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:49.629920 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:35:51.053824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:51.053787 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:35:53.744545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:53.743044 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:53.753548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:53.746487 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:53.766870 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:53.766825 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:53.938563 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:53.924244 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 02 21:35:54.233544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:54.201723 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:54.282550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:54.254606 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.642900 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:54.642838 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Nov 02 21:35:54.755552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:54.747752 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:54.908703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:54.908597 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 02 21:35:54.924510 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:54.924468 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.124055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.010474 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:35:55.124921 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.124897 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:35:55.137660 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.084624 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.435138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.435104 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:35:55.563552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.550888 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 02 21:35:55.604130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.603422 4510 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.605734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.605696 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:35:55.669816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.669774 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 21:35:55.734598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.706950 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:35:55.734598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.731393 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:55.760629 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.759958 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 21:35:55.945678 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.945620 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.006754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.995996 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:35:56.006754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:55.997925 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.149675 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:56.149644 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 21:35:56.151968 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:56.151946 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 21:35:56.152623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:56.123497 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:35:56.159279 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:56.152618 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 21:35:56.324812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:56.324733 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 21:35:56.335549 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:35:56.334987 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:36:15.532512 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:36:15.532431 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:36:20.844585 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.844551 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5"] Nov 02 21:36:20.853716 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.853695 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:20.903951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.903920 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5"] Nov 02 21:36:20.934420 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.934389 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9dd62-cbe7-4063-8e33-46b3840c842f-openshift-service-ca\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:20.934583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.934430 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-metrics-certs\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:20.934665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.934610 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmr4\" (UniqueName: \"kubernetes.io/projected/aff9dd62-cbe7-4063-8e33-46b3840c842f-kube-api-access-vzmr4\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:20.934732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.934668 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:20.944627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:20.944600 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4dff0429-serving-cert\"" Nov 02 21:36:21.036377 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.036343 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmr4\" (UniqueName: \"kubernetes.io/projected/aff9dd62-cbe7-4063-8e33-46b3840c842f-kube-api-access-vzmr4\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.036570 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.036426 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.036662 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:36:21.036614 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-serving-cert: secret "maas-api-e2e-derive-4dff0429-serving-cert" not found Nov 02 21:36:21.036722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.036665 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9dd62-cbe7-4063-8e33-46b3840c842f-openshift-service-ca\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.036722 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:36:21.036692 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls podName:aff9dd62-cbe7-4063-8e33-46b3840c842f nodeName:}" failed. No retries permitted until 2026-11-02 21:36:21.536671802 +0000 UTC m=+1407.512969571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls") pod "maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" (UID: "aff9dd62-cbe7-4063-8e33-46b3840c842f") : secret "maas-api-e2e-derive-4dff0429-serving-cert" not found Nov 02 21:36:21.036838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.036751 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-metrics-certs\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.037373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.037353 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9dd62-cbe7-4063-8e33-46b3840c842f-openshift-service-ca\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.039734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.039717 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-metrics-certs\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.091621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.091590 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmr4\" (UniqueName: \"kubernetes.io/projected/aff9dd62-cbe7-4063-8e33-46b3840c842f-kube-api-access-vzmr4\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.545571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.545505 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.547905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.547881 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls\") pod \"maas-api-e2e-derive-4dff0429-786787cc57-vwsn5\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:21.768074 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:21.768040 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:22.159902 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:22.159875 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5"] Nov 02 21:36:22.519569 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:22.519481 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" event={"ID":"aff9dd62-cbe7-4063-8e33-46b3840c842f","Type":"ContainerStarted","Data":"863d33f3652b288ecf5997cef9347fccac4a2f29a344efcb4b4d50b7de6c9e51"} Nov 02 21:36:23.527724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:23.527687 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" event={"ID":"aff9dd62-cbe7-4063-8e33-46b3840c842f","Type":"ContainerStarted","Data":"0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4"} Nov 02 21:36:23.528113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:23.527754 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:23.637030 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:23.636978 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" podStartSLOduration=2.740967324 podStartE2EDuration="3.636964976s" totalImagesPullingTime="895.997652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:23.632732854 +0000 UTC m=+1409.609030633" watchObservedRunningTime="2026-11-02 21:36:23.636964976 +0000 UTC m=+1409.613262750" Nov 02 21:36:27.032545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.032482 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg"] Nov 02 21:36:27.037307 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.037286 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.133655 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.133611 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.133864 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.133702 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.133864 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.133757 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.133864 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.133776 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd937071-bddd-46ae-b7a6-1a8523442445-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.133864 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.133801 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd937071-bddd-46ae-b7a6-1a8523442445-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.134104 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.133983 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.134161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.134136 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xpr\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-kube-api-access-b9xpr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.134272 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.134252 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.134338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.134317 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.188726 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.188679 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-445mt\"" Nov 02 21:36:27.234718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.234684 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg"] Nov 02 21:36:27.235618 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.235590 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xpr\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-kube-api-access-b9xpr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.235814 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.235791 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.235905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.235849 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236050 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236167 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236097 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236167 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236162 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236283 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236187 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd937071-bddd-46ae-b7a6-1a8523442445-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236283 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236214 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd937071-bddd-46ae-b7a6-1a8523442445-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236274 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236295 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236331 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236585 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236411 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.236948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236927 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd937071-bddd-46ae-b7a6-1a8523442445-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.237033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.236974 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.238760 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.238721 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.239157 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.239137 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd937071-bddd-46ae-b7a6-1a8523442445-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.288885 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.288807 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.289280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.289260 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xpr\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-kube-api-access-b9xpr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sz8lg\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.352201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.352173 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:27.784720 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.784689 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg"] Nov 02 21:36:27.790209 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:36:27.790185 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd937071_bddd_46ae_b7a6_1a8523442445.slice/crio-87e33a4fa0cb3eedd3f5ccdb5b3cd2621deccea0c8b896bf5ab20dfaa1b70017 WatchSource:0}: Error finding container 87e33a4fa0cb3eedd3f5ccdb5b3cd2621deccea0c8b896bf5ab20dfaa1b70017: Status 404 returned error can't find the container with id 87e33a4fa0cb3eedd3f5ccdb5b3cd2621deccea0c8b896bf5ab20dfaa1b70017 Nov 02 21:36:27.792807 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.792774 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:27.792897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.792838 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:27.792897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:27.792871 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:28.578238 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:28.578191 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" event={"ID":"cd937071-bddd-46ae-b7a6-1a8523442445","Type":"ContainerStarted","Data":"a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff"} Nov 02 21:36:28.578789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:28.578313 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" event={"ID":"cd937071-bddd-46ae-b7a6-1a8523442445","Type":"ContainerStarted","Data":"87e33a4fa0cb3eedd3f5ccdb5b3cd2621deccea0c8b896bf5ab20dfaa1b70017"} Nov 02 21:36:28.790631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:28.790580 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podStartSLOduration=2.7905669189999998 podStartE2EDuration="2.790566919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:28.74104799 +0000 UTC m=+1414.717345778" watchObservedRunningTime="2026-11-02 21:36:28.790566919 +0000 UTC m=+1414.766864694" Nov 02 21:36:29.354565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:29.353285 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:29.354768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:29.354680 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:36:29.354768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:29.354750 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:36:29.553295 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:29.553240 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:36:30.353177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:30.353137 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:36:30.353611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:30.353204 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:36:31.353315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:31.353152 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:36:31.353315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:31.353215 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:36:32.353488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:32.353450 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:36:32.353911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:32.353565 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:36:33.353024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:33.352986 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:36:33.353254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:33.353055 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:36:34.352754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:34.352701 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:36:34.353324 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:34.352761 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:36:34.877134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:34.877102 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq"] Nov 02 21:36:34.882191 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:34.882170 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:34.977108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:34.977070 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c7cd2d37-serving-cert\"" Nov 02 21:36:34.984500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:34.984464 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq"] Nov 02 21:36:35.028329 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.028285 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-metrics-certs\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.028672 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.028646 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-maas-api-tls\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.028915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.028895 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpcj\" (UniqueName: \"kubernetes.io/projected/3f335f26-4d8c-4fa2-b748-7a877375adb1-kube-api-access-zgpcj\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.029253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.029233 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335f26-4d8c-4fa2-b748-7a877375adb1-openshift-service-ca\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.133214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.132018 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-metrics-certs\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.133214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.132085 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-maas-api-tls\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.133214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.132145 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpcj\" (UniqueName: \"kubernetes.io/projected/3f335f26-4d8c-4fa2-b748-7a877375adb1-kube-api-access-zgpcj\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.133214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.132444 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335f26-4d8c-4fa2-b748-7a877375adb1-openshift-service-ca\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.135683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.135652 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335f26-4d8c-4fa2-b748-7a877375adb1-openshift-service-ca\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.137473 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.137451 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-maas-api-tls\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.137703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.137682 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-metrics-certs\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.187205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.187167 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpcj\" (UniqueName: \"kubernetes.io/projected/3f335f26-4d8c-4fa2-b748-7a877375adb1-kube-api-access-zgpcj\") pod \"maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.197769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.197726 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:35.356766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.356726 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 21:36:35.357270 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.356788 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 21:36:35.589710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.589619 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq"] Nov 02 21:36:35.596921 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:36:35.596888 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f335f26_4d8c_4fa2_b748_7a877375adb1.slice/crio-7317b6020fca86c64e9329d3bf86ba3d8289379f8ebc8dda5587d3cc8ccb9f23 WatchSource:0}: Error finding container 7317b6020fca86c64e9329d3bf86ba3d8289379f8ebc8dda5587d3cc8ccb9f23: Status 404 returned error can't find the container with id 7317b6020fca86c64e9329d3bf86ba3d8289379f8ebc8dda5587d3cc8ccb9f23 Nov 02 21:36:35.630152 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:35.630124 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" event={"ID":"3f335f26-4d8c-4fa2-b748-7a877375adb1","Type":"ContainerStarted","Data":"7317b6020fca86c64e9329d3bf86ba3d8289379f8ebc8dda5587d3cc8ccb9f23"} Nov 02 21:36:36.642184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:36.642146 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" event={"ID":"3f335f26-4d8c-4fa2-b748-7a877375adb1","Type":"ContainerStarted","Data":"9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b"} Nov 02 21:36:36.642722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:36.642238 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:36.756610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:36.756470 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" podStartSLOduration=1.901290367 podStartE2EDuration="2.75645543s" totalImagesPullingTime="855.165063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:36:36.749385435 +0000 UTC m=+1422.725683213" watchObservedRunningTime="2026-11-02 21:36:36.75645543 +0000 UTC m=+1422.732753204" Nov 02 21:36:36.790234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:36.790201 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:36.791721 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:36.791697 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:36.792560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:36.792511 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 21:36:42.654717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:42.654688 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:36:45.537073 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:36:45.537030 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:36:47.735149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:47.735074 4510 generic.go:356] "Generic (PLEG): container finished" podID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerID="0a73413e0f0e8d7a3c421b8ecd52f68867b5f0133cec5c985b8c0efb3c2c687c" exitCode=0 Nov 02 21:36:47.735619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:47.735147 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" event={"ID":"888afd07-c4f2-4772-89bc-f6a9f3408041","Type":"ContainerDied","Data":"0a73413e0f0e8d7a3c421b8ecd52f68867b5f0133cec5c985b8c0efb3c2c687c"} Nov 02 21:36:47.735619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:47.735580 4510 scope.go:122] "RemoveContainer" containerID="0a73413e0f0e8d7a3c421b8ecd52f68867b5f0133cec5c985b8c0efb3c2c687c" Nov 02 21:36:47.737091 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:47.737061 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:47.737159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:47.737132 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:47.737207 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:47.737165 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:36:48.755543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:48.751948 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" event={"ID":"888afd07-c4f2-4772-89bc-f6a9f3408041","Type":"ContainerStarted","Data":"da7fed1c4432b14fed1ea9159e1ad67be0fe4a1cf69204a1352c8292982ab162"} Nov 02 21:36:49.384851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:49.384820 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:36:49.385888 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:49.385864 4510 patch_prober.go:29] interesting pod/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 21:36:49.385992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:49.385915 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 21:36:50.385143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:50.385105 4510 patch_prober.go:29] interesting pod/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 21:36:50.385575 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:50.385163 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 21:36:51.384993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:51.384958 4510 patch_prober.go:29] interesting pod/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 21:36:51.385224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:51.385018 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 21:36:52.368744 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:52.368715 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:36:52.384439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:52.384411 4510 patch_prober.go:29] interesting pod/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 21:36:52.384570 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:52.384458 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 21:36:53.384281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:53.384247 4510 patch_prober.go:29] interesting pod/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 21:36:53.384711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:53.384300 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 21:36:54.388838 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:54.388802 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:36:54.389931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:36:54.389911 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:37:12.361078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.361043 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh"] Nov 02 21:37:12.367362 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.367341 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.470353 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.470316 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh"] Nov 02 21:37:12.540394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.540359 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.540609 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.540502 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjj8p\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-kube-api-access-tjj8p\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.540800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.540759 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.540870 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.540819 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.540931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.540885 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.540931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.540910 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c508601-c5f7-440f-b605-fb035d963673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.541049 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.540997 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.541049 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.541030 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.541152 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.541054 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c508601-c5f7-440f-b605-fb035d963673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.644928 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.644892 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.644928 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.644936 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.644991 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.645015 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c508601-c5f7-440f-b605-fb035d963673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.645097 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.645151 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645313 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.645190 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c508601-c5f7-440f-b605-fb035d963673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645313 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.645257 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.645342 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.645392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.645362 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjj8p\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-kube-api-access-tjj8p\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.647593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.647489 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.647750 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.647664 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.647903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.647832 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.648328 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.648275 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c508601-c5f7-440f-b605-fb035d963673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.654200 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.654178 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.654366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.654329 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c508601-c5f7-440f-b605-fb035d963673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.698931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.698894 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.699361 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.699340 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjj8p\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-kube-api-access-tjj8p\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lq8nh\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:12.981781 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:12.981693 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:13.416164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:13.416128 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh"] Nov 02 21:37:13.422281 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:37:13.422253 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c508601_c5f7_440f_b605_fb035d963673.slice/crio-ab8fe69ed138a5de641bb2f007c69de1a26234ed1218c8fa7fd03d8d3769683f WatchSource:0}: Error finding container ab8fe69ed138a5de641bb2f007c69de1a26234ed1218c8fa7fd03d8d3769683f: Status 404 returned error can't find the container with id ab8fe69ed138a5de641bb2f007c69de1a26234ed1218c8fa7fd03d8d3769683f Nov 02 21:37:13.424903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:13.424867 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:13.424993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:13.424934 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:13.424993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:13.424962 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:13.948293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:13.948258 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" event={"ID":"1c508601-c5f7-440f-b605-fb035d963673","Type":"ContainerStarted","Data":"8f35793f12da742476be6263f2e0269b9ef00f9d6b79fafa4f7e53b8e4fadf50"} Nov 02 21:37:13.948479 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:13.948317 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" event={"ID":"1c508601-c5f7-440f-b605-fb035d963673","Type":"ContainerStarted","Data":"ab8fe69ed138a5de641bb2f007c69de1a26234ed1218c8fa7fd03d8d3769683f"} Nov 02 21:37:13.982371 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:13.982343 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:14.101853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:14.101798 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podStartSLOduration=2.101766366 podStartE2EDuration="2.101766366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:14.10101858 +0000 UTC m=+1460.077316356" watchObservedRunningTime="2026-11-02 21:37:14.101766366 +0000 UTC m=+1460.078064141" Nov 02 21:37:14.984658 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:14.984624 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:37:14.985057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:14.984683 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 21:37:15.553219 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:37:15.553181 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:37:15.982793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:15.982761 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:37:15.982964 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:15.982822 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 21:37:16.982148 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:16.982117 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:37:16.982579 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:16.982172 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 21:37:17.982856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:17.982823 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:37:17.983250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:17.982882 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 21:37:18.982732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:18.982696 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:37:18.982936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:18.982754 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 21:37:19.983015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:19.982970 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:37:19.983449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:19.983054 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 21:37:20.474742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:20.474709 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:37:20.982911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:20.982869 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 21:37:20.983098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:20.982938 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 21:37:22.983512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:22.983167 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:37:22.983512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:22.983225 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:37:23.042587 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:23.041604 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:23.043056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:23.043034 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:23.043815 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:23.043793 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:37:42.593896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.593812 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k"] Nov 02 21:37:42.598940 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.598921 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.678318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.678286 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.678605 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.678582 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.678708 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.678621 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.678793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.678765 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.678835 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.678818 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.678902 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.678882 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.678969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.678956 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l27f\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-kube-api-access-9l27f\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.679068 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.679052 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.679212 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.679190 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.701713 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.701684 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k"] Nov 02 21:37:42.780900 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.780870 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.780932 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.780958 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781011 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781037 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781056 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781117 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l27f\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-kube-api-access-9l27f\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781175 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781283 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781322 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781305 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781423 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.781689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.781603 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.782113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.782093 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.783581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.783556 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.783688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.783662 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.833765 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.833735 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.833915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.833899 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l27f\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-kube-api-access-9l27f\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-n2h8k\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:42.914330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:42.914308 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:43.332241 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:43.332160 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k"] Nov 02 21:37:43.337748 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:37:43.337723 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9632c7_52f5_4223_aeb4_4b2fcef13522.slice/crio-b1010149957169add564b40f438fb2aec20bc2f29fc16808adc0d97d955bad32 WatchSource:0}: Error finding container b1010149957169add564b40f438fb2aec20bc2f29fc16808adc0d97d955bad32: Status 404 returned error can't find the container with id b1010149957169add564b40f438fb2aec20bc2f29fc16808adc0d97d955bad32 Nov 02 21:37:43.340188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:43.340156 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:43.340280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:43.340241 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:43.340280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:43.340269 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:37:44.213580 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:44.213543 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" event={"ID":"0c9632c7-52f5-4223-aeb4-4b2fcef13522","Type":"ContainerStarted","Data":"f34cc388568f418d2e72e068439e1f6a9bbec26244ec96a35b22b2edd8ff65e9"} Nov 02 21:37:44.214064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:44.213596 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" event={"ID":"0c9632c7-52f5-4223-aeb4-4b2fcef13522","Type":"ContainerStarted","Data":"b1010149957169add564b40f438fb2aec20bc2f29fc16808adc0d97d955bad32"} Nov 02 21:37:44.419972 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:44.419911 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podStartSLOduration=2.419895699 podStartE2EDuration="2.419895699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:37:44.370040768 +0000 UTC m=+1490.346338543" watchObservedRunningTime="2026-11-02 21:37:44.419895699 +0000 UTC m=+1490.396193474" Nov 02 21:37:44.915556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:44.915498 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:44.916988 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:44.916960 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:44.917108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:44.917012 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:45.557326 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:37:45.557288 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:37:45.915266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:45.915231 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:45.915432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:45.915300 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:46.914926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:46.914892 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:46.915374 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:46.914948 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:47.915504 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:47.915425 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:47.915504 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:47.915486 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:48.915555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:48.915487 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:48.916028 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:48.915560 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:49.915588 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:49.915537 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:49.916024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:49.915603 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:50.915444 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:50.915412 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 21:37:50.915769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:50.915468 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 21:37:51.920333 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:51.920300 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:51.920859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:51.920647 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:37:51.921349 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:37:51.921327 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:38:04.786507 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:04.786467 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb"] Nov 02 21:38:04.787065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:04.786816 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" containerID="cri-o://da7fed1c4432b14fed1ea9159e1ad67be0fe4a1cf69204a1352c8292982ab162" gracePeriod=30 Nov 02 21:38:09.132155 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.132119 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5"] Nov 02 21:38:09.132590 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.132354 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" podUID="aff9dd62-cbe7-4063-8e33-46b3840c842f" containerName="maas-api" containerID="cri-o://0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4" gracePeriod=30 Nov 02 21:38:09.402229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.402125 4510 patch_prober.go:29] interesting pod/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:38:09.402229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.402214 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:38:09.474096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.474038 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:38:09.499263 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.499228 4510 generic.go:356] "Generic (PLEG): container finished" podID="aff9dd62-cbe7-4063-8e33-46b3840c842f" containerID="0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4" exitCode=0 Nov 02 21:38:09.499416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.499279 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" Nov 02 21:38:09.499416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.499299 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" event={"ID":"aff9dd62-cbe7-4063-8e33-46b3840c842f","Type":"ContainerDied","Data":"0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4"} Nov 02 21:38:09.499416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.499353 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5" event={"ID":"aff9dd62-cbe7-4063-8e33-46b3840c842f","Type":"ContainerDied","Data":"863d33f3652b288ecf5997cef9347fccac4a2f29a344efcb4b4d50b7de6c9e51"} Nov 02 21:38:09.499416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.499374 4510 scope.go:122] "RemoveContainer" containerID="0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4" Nov 02 21:38:09.511735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.511712 4510 scope.go:122] "RemoveContainer" containerID="0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4" Nov 02 21:38:09.512066 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:38:09.512041 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4\": container with ID starting with 0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4 not found: ID does not exist" containerID="0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4" Nov 02 21:38:09.512189 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.512072 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4"} err="failed to get container status \"0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4\": rpc error: code = NotFound desc = could not find container \"0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4\": container with ID starting with 0bb1991184073486a5e39aafcb227e36406799e7dcaacc140bcb260342832ca4 not found: ID does not exist" Nov 02 21:38:09.536032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.535996 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff9dd62-cbe7-4063-8e33-46b3840c842f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9dd62-cbe7-4063-8e33-46b3840c842f-openshift-service-ca\") pod \"aff9dd62-cbe7-4063-8e33-46b3840c842f\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " Nov 02 21:38:09.536248 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.536165 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-metrics-certs\") pod \"aff9dd62-cbe7-4063-8e33-46b3840c842f\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " Nov 02 21:38:09.536338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.536294 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls\") pod \"aff9dd62-cbe7-4063-8e33-46b3840c842f\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " Nov 02 21:38:09.536395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.536357 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff9dd62-cbe7-4063-8e33-46b3840c842f-kube-api-access-vzmr4\" (UniqueName: \"kubernetes.io/projected/aff9dd62-cbe7-4063-8e33-46b3840c842f-kube-api-access-vzmr4\") pod \"aff9dd62-cbe7-4063-8e33-46b3840c842f\" (UID: \"aff9dd62-cbe7-4063-8e33-46b3840c842f\") " Nov 02 21:38:09.536449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.536388 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff9dd62-cbe7-4063-8e33-46b3840c842f-openshift-service-ca" pod "aff9dd62-cbe7-4063-8e33-46b3840c842f" (UID: "aff9dd62-cbe7-4063-8e33-46b3840c842f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:09.537240 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.537211 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff9dd62-cbe7-4063-8e33-46b3840c842f-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:09.538831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.538806 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-metrics-certs" pod "aff9dd62-cbe7-4063-8e33-46b3840c842f" (UID: "aff9dd62-cbe7-4063-8e33-46b3840c842f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:38:09.539107 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.539078 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff9dd62-cbe7-4063-8e33-46b3840c842f-kube-api-access-vzmr4" pod "aff9dd62-cbe7-4063-8e33-46b3840c842f" (UID: "aff9dd62-cbe7-4063-8e33-46b3840c842f"). InnerVolumeSpecName "kube-api-access-vzmr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:09.539303 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.539282 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls" pod "aff9dd62-cbe7-4063-8e33-46b3840c842f" (UID: "aff9dd62-cbe7-4063-8e33-46b3840c842f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:38:09.659985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.659878 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:09.659985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.659916 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff9dd62-cbe7-4063-8e33-46b3840c842f-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:09.659985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.659933 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzmr4\" (UniqueName: \"kubernetes.io/projected/aff9dd62-cbe7-4063-8e33-46b3840c842f-kube-api-access-vzmr4\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:09.974796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.974368 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5"] Nov 02 21:38:10.007542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:09.999436 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4dff0429-786787cc57-vwsn5"] Nov 02 21:38:10.167542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.167326 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff9dd62-cbe7-4063-8e33-46b3840c842f" path="/var/lib/kubelet/pods/aff9dd62-cbe7-4063-8e33-46b3840c842f/volumes" Nov 02 21:38:10.538323 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.538123 4510 generic.go:356] "Generic (PLEG): container finished" podID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerID="da7fed1c4432b14fed1ea9159e1ad67be0fe4a1cf69204a1352c8292982ab162" exitCode=0 Nov 02 21:38:10.538323 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.538200 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" event={"ID":"888afd07-c4f2-4772-89bc-f6a9f3408041","Type":"ContainerDied","Data":"da7fed1c4432b14fed1ea9159e1ad67be0fe4a1cf69204a1352c8292982ab162"} Nov 02 21:38:10.538323 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.538255 4510 scope.go:122] "RemoveContainer" containerID="0a73413e0f0e8d7a3c421b8ecd52f68867b5f0133cec5c985b8c0efb3c2c687c" Nov 02 21:38:10.606482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.606453 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:38:10.686581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686508 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-credential-socket\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.686745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686644 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-podinfo\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.686745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686689 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-kube-api-access-82zlb\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-kube-api-access-82zlb\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.686745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686736 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-data\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.686915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686768 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/888afd07-c4f2-4772-89bc-f6a9f3408041-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/888afd07-c4f2-4772-89bc-f6a9f3408041-istiod-ca-cert\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.686915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686773 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-credential-socket" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.686915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686880 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-certs\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.687080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.686915 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-envoy\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.687080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687030 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-socket\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.687080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687045 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-data" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.687236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687099 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-token\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-token\") pod \"888afd07-c4f2-4772-89bc-f6a9f3408041\" (UID: \"888afd07-c4f2-4772-89bc-f6a9f3408041\") " Nov 02 21:38:10.687236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687153 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/888afd07-c4f2-4772-89bc-f6a9f3408041-istiod-ca-cert" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:38:10.687334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687287 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-socket" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.687378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687339 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-certs" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.687860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687837 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.687860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687858 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/888afd07-c4f2-4772-89bc-f6a9f3408041-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.688032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687868 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.688032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687876 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.688032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.687885 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.689034 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.689009 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-kube-api-access-82zlb" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "kube-api-access-82zlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:10.689133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.689110 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-token" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:38:10.689280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.689261 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-podinfo" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:38:10.689342 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.689265 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-envoy" pod "888afd07-c4f2-4772-89bc-f6a9f3408041" (UID: "888afd07-c4f2-4772-89bc-f6a9f3408041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:38:10.789910 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.789878 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.789910 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.789909 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82zlb\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-kube-api-access-82zlb\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.790100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.789924 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:10.790100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:10.789936 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/888afd07-c4f2-4772-89bc-f6a9f3408041-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:38:11.547080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:11.547049 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" Nov 02 21:38:11.547554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:11.547094 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb" event={"ID":"888afd07-c4f2-4772-89bc-f6a9f3408041","Type":"ContainerDied","Data":"d8f991cb62dae6550fa5c4b088fe275fa4cf9ee766eec4010a949470927ac341"} Nov 02 21:38:11.547554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:11.547160 4510 scope.go:122] "RemoveContainer" containerID="da7fed1c4432b14fed1ea9159e1ad67be0fe4a1cf69204a1352c8292982ab162" Nov 02 21:38:11.702266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:11.702228 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb"] Nov 02 21:38:11.706207 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:11.706181 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2ccb9fb8-gw-openshift-default-bd688b445-9vzqb"] Nov 02 21:38:12.156606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:12.156566 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" path="/var/lib/kubelet/pods/888afd07-c4f2-4772-89bc-f6a9f3408041/volumes" Nov 02 21:38:15.531670 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:38:15.531628 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:38:22.326951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:22.326913 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:38:45.535451 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:38:45.535415 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:38:57.766291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.766253 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq"] Nov 02 21:38:57.767453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767429 4510 cpu_manager.go:414] "containerMap: removing container" podUID="aff9dd62-cbe7-4063-8e33-46b3840c842f" containerName="maas-api" Nov 02 21:38:57.767453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767454 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff9dd62-cbe7-4063-8e33-46b3840c842f" containerName="maas-api" Nov 02 21:38:57.767611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767486 4510 cpu_manager.go:414] "containerMap: removing container" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" Nov 02 21:38:57.767611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767494 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" Nov 02 21:38:57.767611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767539 4510 cpu_manager.go:414] "containerMap: removing container" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" Nov 02 21:38:57.767611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767549 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" Nov 02 21:38:57.767753 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767710 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" Nov 02 21:38:57.767753 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767724 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="888afd07-c4f2-4772-89bc-f6a9f3408041" containerName="istio-proxy" Nov 02 21:38:57.767753 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.767744 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff9dd62-cbe7-4063-8e33-46b3840c842f" containerName="maas-api" Nov 02 21:38:57.773821 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.773803 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:57.824171 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.824144 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq"] Nov 02 21:38:57.865315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.865278 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e942516f-serving-cert\"" Nov 02 21:38:57.922021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.921983 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cst\" (UniqueName: \"kubernetes.io/projected/bde135ac-2ebd-4377-a7fd-5aed73856855-kube-api-access-b2cst\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:57.922244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.922222 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-metrics-certs\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:57.922414 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.922307 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-maas-api-tls\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:57.922534 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:57.922426 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde135ac-2ebd-4377-a7fd-5aed73856855-openshift-service-ca\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.033392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.032563 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-metrics-certs\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.033623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.033415 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-maas-api-tls\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.033792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.033764 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde135ac-2ebd-4377-a7fd-5aed73856855-openshift-service-ca\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.034038 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.033934 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cst\" (UniqueName: \"kubernetes.io/projected/bde135ac-2ebd-4377-a7fd-5aed73856855-kube-api-access-b2cst\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.035359 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.035332 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde135ac-2ebd-4377-a7fd-5aed73856855-openshift-service-ca\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.044815 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.040056 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-maas-api-tls\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.046086 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.046034 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-metrics-certs\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.087262 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.087225 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cst\" (UniqueName: \"kubernetes.io/projected/bde135ac-2ebd-4377-a7fd-5aed73856855-kube-api-access-b2cst\") pod \"maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.088997 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.088973 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:38:58.498646 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.498610 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq"] Nov 02 21:38:58.505984 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:38:58.505954 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde135ac_2ebd_4377_a7fd_5aed73856855.slice/crio-acf23147e642b4623b090cb837a0dd534c0c34e78757dd612ddd783b60562051 WatchSource:0}: Error finding container acf23147e642b4623b090cb837a0dd534c0c34e78757dd612ddd783b60562051: Status 404 returned error can't find the container with id acf23147e642b4623b090cb837a0dd534c0c34e78757dd612ddd783b60562051 Nov 02 21:38:58.960742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:58.960708 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" event={"ID":"bde135ac-2ebd-4377-a7fd-5aed73856855","Type":"ContainerStarted","Data":"acf23147e642b4623b090cb837a0dd534c0c34e78757dd612ddd783b60562051"} Nov 02 21:38:59.969784 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:59.969750 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" event={"ID":"bde135ac-2ebd-4377-a7fd-5aed73856855","Type":"ContainerStarted","Data":"d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4"} Nov 02 21:38:59.970182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:38:59.969844 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:39:00.084299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:00.084241 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" podStartSLOduration=2.204243635 podStartE2EDuration="3.084222787s" totalImagesPullingTime="879.979152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:00.078733369 +0000 UTC m=+1566.055031142" watchObservedRunningTime="2026-11-02 21:39:00.084222787 +0000 UTC m=+1566.060520569" Nov 02 21:39:05.985153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:05.985081 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:39:15.532042 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:39:15.532004 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:39:19.654637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:19.653691 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:39:19.788049 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:19.788006 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5"] Nov 02 21:39:19.788268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:19.788240 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" containerID="cri-o://ca02590464fa2805068da9bd7f82a7179031f1d754f3a0e1fc91efbf26a2ab90" gracePeriod=30 Nov 02 21:39:20.125081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:20.125051 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw"] Nov 02 21:39:20.125303 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:20.125279 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" containerID="cri-o://61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69" gracePeriod=30 Nov 02 21:39:20.130360 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:20.130283 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb"] Nov 02 21:39:20.130669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:20.130638 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" containerID="cri-o://c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54" gracePeriod=30 Nov 02 21:39:20.136829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:20.136789 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx"] Nov 02 21:39:20.145581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:20.137566 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" containerID="cri-o://822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2" gracePeriod=30 Nov 02 21:39:23.152692 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:23.152657 4510 patch_prober.go:29] interesting pod/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:39:23.153098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:23.152721 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:39:25.173783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.173748 4510 generic.go:356] "Generic (PLEG): container finished" podID="87732d55-1a01-4d5d-b699-19edbff59b35" containerID="ca02590464fa2805068da9bd7f82a7179031f1d754f3a0e1fc91efbf26a2ab90" exitCode=0 Nov 02 21:39:25.174163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.173886 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" event={"ID":"87732d55-1a01-4d5d-b699-19edbff59b35","Type":"ContainerDied","Data":"ca02590464fa2805068da9bd7f82a7179031f1d754f3a0e1fc91efbf26a2ab90"} Nov 02 21:39:25.174163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.173968 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" event={"ID":"87732d55-1a01-4d5d-b699-19edbff59b35","Type":"ContainerDied","Data":"b3b2a48ffbed287e96faaf72f84b18e1c69a0c1b3eee0db4d432452d90c9c031"} Nov 02 21:39:25.174163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.174005 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b2a48ffbed287e96faaf72f84b18e1c69a0c1b3eee0db4d432452d90c9c031" Nov 02 21:39:25.328720 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.328112 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:39:25.469026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.468999 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:39:25.476300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.476270 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-kube-api-access-5kr45\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-kube-api-access-5kr45\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.476576 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.476550 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-data\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.476670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.476649 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87732d55-1a01-4d5d-b699-19edbff59b35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87732d55-1a01-4d5d-b699-19edbff59b35-istio-podinfo\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.476772 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.476752 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-data" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.476999 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.476980 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-socket" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.477084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477062 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-socket\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.477147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477134 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-certs\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.477233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477216 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-envoy\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.477297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477251 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-data\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.477371 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477355 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87732d55-1a01-4d5d-b699-19edbff59b35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87732d55-1a01-4d5d-b699-19edbff59b35-istiod-ca-cert\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.477428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477418 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-credential-socket\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.477784 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477762 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-certs\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.477877 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477836 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-istio-token\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-istio-token\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.478154 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.477973 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-certs" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.478736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.478395 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87732d55-1a01-4d5d-b699-19edbff59b35-istiod-ca-cert" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:25.479115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.478936 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-data" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.479434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.479267 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-credential-socket" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.479434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.479372 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-socket\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.479434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.479409 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istiod-ca-cert\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.479663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.479462 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-envoy\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.479663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.479503 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-credential-socket\") pod \"87732d55-1a01-4d5d-b699-19edbff59b35\" (UID: \"87732d55-1a01-4d5d-b699-19edbff59b35\") " Nov 02 21:39:25.479663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.479613 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-podinfo\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.479663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.479657 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-kube-api-access-mjcfl\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-kube-api-access-mjcfl\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480077 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-certs" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480201 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istiod-ca-cert" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480501 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480537 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480553 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87732d55-1a01-4d5d-b699-19edbff59b35-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480567 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480580 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480594 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480607 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.480642 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480620 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.481231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.480888 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-credential-socket" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.481231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.481123 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-socket" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.481341 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.481235 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-envoy" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.483076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.483050 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87732d55-1a01-4d5d-b699-19edbff59b35-istio-podinfo" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:39:25.492109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.492032 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-kube-api-access-5kr45" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "kube-api-access-5kr45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.492109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.492074 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-istio-token" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.493683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.493660 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-envoy" pod "87732d55-1a01-4d5d-b699-19edbff59b35" (UID: "87732d55-1a01-4d5d-b699-19edbff59b35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.494613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.494590 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-kube-api-access-mjcfl" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "kube-api-access-mjcfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.495415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.495395 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-podinfo" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:39:25.535835 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.535811 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:39:25.582294 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582250 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-data\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582315 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-credential-socket\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582424 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-token\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-token\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582460 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-token\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-token\") pod \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\" (UID: \"f26a9ed3-7950-4ad2-957d-0deb5beb99c0\") " Nov 02 21:39:25.582599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582515 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-certs\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582598 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-data" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.582738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582643 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-kube-api-access-sm5hr\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-kube-api-access-sm5hr\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582690 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-credential-socket" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.582738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582713 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-podinfo\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582748 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istiod-ca-cert\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582808 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-envoy\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.582966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582901 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-certs" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.582966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.582935 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-socket\") pod \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\" (UID: \"17a339a5-da47-4fdc-9cf5-7d5ba07b735d\") " Nov 02 21:39:25.583635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583580 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-socket" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.583635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583601 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583621 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583636 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583650 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583665 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87732d55-1a01-4d5d-b699-19edbff59b35-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583671 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istiod-ca-cert" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583681 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583719 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583736 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583754 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjcfl\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-kube-api-access-mjcfl\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583773 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kr45\" (UniqueName: \"kubernetes.io/projected/87732d55-1a01-4d5d-b699-19edbff59b35-kube-api-access-5kr45\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583788 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.583915 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.583801 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87732d55-1a01-4d5d-b699-19edbff59b35-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.585508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.585478 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-token" pod "f26a9ed3-7950-4ad2-957d-0deb5beb99c0" (UID: "f26a9ed3-7950-4ad2-957d-0deb5beb99c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.585863 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.585834 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-kube-api-access-sm5hr" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "kube-api-access-sm5hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.585978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.585960 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-podinfo" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:39:25.586373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.586350 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-token" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.587047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.587024 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-envoy" pod "17a339a5-da47-4fdc-9cf5-7d5ba07b735d" (UID: "17a339a5-da47-4fdc-9cf5-7d5ba07b735d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.685550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.685469 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.685550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.685498 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.685550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.685508 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.685550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.685549 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26a9ed3-7950-4ad2-957d-0deb5beb99c0-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.685767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.685560 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm5hr\" (UniqueName: \"kubernetes.io/projected/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-kube-api-access-sm5hr\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.685767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.685571 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.685767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.685579 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a339a5-da47-4fdc-9cf5-7d5ba07b735d-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.859132 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.859105 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:39:25.890453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890412 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-data\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.890669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890506 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-credential-socket\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.890669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890591 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-token\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-token\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.890669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890633 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b866e72-25c8-4171-ad22-f5eff84395b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b866e72-25c8-4171-ad22-f5eff84395b7-istiod-ca-cert\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.890669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890666 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-socket\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.890868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890777 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-credential-socket" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.890868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890803 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-data" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.890868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890835 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-podinfo\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.891023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890936 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-envoy\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.891023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.890978 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-certs\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.891134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891032 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b866e72-25c8-4171-ad22-f5eff84395b7-istiod-ca-cert" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:39:25.891134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891038 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-socket" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.891134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891075 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-kube-api-access-srdf6\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-kube-api-access-srdf6\") pod \"5b866e72-25c8-4171-ad22-f5eff84395b7\" (UID: \"5b866e72-25c8-4171-ad22-f5eff84395b7\") " Nov 02 21:39:25.891301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891280 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-certs" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.891837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891812 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.891980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891840 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.891980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891903 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.891980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891942 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b866e72-25c8-4171-ad22-f5eff84395b7-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.891980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.891956 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.893374 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.893345 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-token" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.893586 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.893562 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-podinfo" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:39:25.893683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.893664 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-envoy" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:39:25.893731 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.893682 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-kube-api-access-srdf6" pod "5b866e72-25c8-4171-ad22-f5eff84395b7" (UID: "5b866e72-25c8-4171-ad22-f5eff84395b7"). InnerVolumeSpecName "kube-api-access-srdf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:39:25.993818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.993729 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.993818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.993757 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.993818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.993766 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b866e72-25c8-4171-ad22-f5eff84395b7-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:25.993818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:25.993775 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srdf6\" (UniqueName: \"kubernetes.io/projected/5b866e72-25c8-4171-ad22-f5eff84395b7-kube-api-access-srdf6\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:39:26.181565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.181540 4510 generic.go:356] "Generic (PLEG): container finished" podID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerID="c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54" exitCode=0 Nov 02 21:39:26.181895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.181635 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" Nov 02 21:39:26.181895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.181673 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" event={"ID":"5b866e72-25c8-4171-ad22-f5eff84395b7","Type":"ContainerDied","Data":"c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54"} Nov 02 21:39:26.181895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.181724 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb" event={"ID":"5b866e72-25c8-4171-ad22-f5eff84395b7","Type":"ContainerDied","Data":"163075d63d64601883c4f26621300b858b3e0de417bf5de0ea7e05c8ed522313"} Nov 02 21:39:26.181895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.181751 4510 scope.go:122] "RemoveContainer" containerID="c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54" Nov 02 21:39:26.183294 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.183268 4510 generic.go:356] "Generic (PLEG): container finished" podID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerID="61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69" exitCode=0 Nov 02 21:39:26.183395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.183315 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" Nov 02 21:39:26.183395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.183363 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" event={"ID":"f26a9ed3-7950-4ad2-957d-0deb5beb99c0","Type":"ContainerDied","Data":"61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69"} Nov 02 21:39:26.183592 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.183443 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw" event={"ID":"f26a9ed3-7950-4ad2-957d-0deb5beb99c0","Type":"ContainerDied","Data":"fd4881a49ac62a610e993d475560f14d232b8e2e3c449ad152057a8f9dbf8fdf"} Nov 02 21:39:26.185115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.185086 4510 generic.go:356] "Generic (PLEG): container finished" podID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerID="822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2" exitCode=0 Nov 02 21:39:26.185221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.185124 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" Nov 02 21:39:26.185221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.185165 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" event={"ID":"17a339a5-da47-4fdc-9cf5-7d5ba07b735d","Type":"ContainerDied","Data":"822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2"} Nov 02 21:39:26.185221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.185200 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx" event={"ID":"17a339a5-da47-4fdc-9cf5-7d5ba07b735d","Type":"ContainerDied","Data":"cb3d5f329409b912d773b1e7d8d01437ae3ee39f5a6cf2a2e4aa1d17a3b459bf"} Nov 02 21:39:26.185380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.185313 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5" Nov 02 21:39:26.198679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.198663 4510 scope.go:122] "RemoveContainer" containerID="c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54" Nov 02 21:39:26.198971 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:39:26.198952 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54\": container with ID starting with c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54 not found: ID does not exist" containerID="c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54" Nov 02 21:39:26.199041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.198980 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54"} err="failed to get container status \"c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54\": rpc error: code = NotFound desc = could not find container \"c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54\": container with ID starting with c960d7707b15de45cfa21fdfe270f36a2b7a4635ebbedb63b2826e73c70abc54 not found: ID does not exist" Nov 02 21:39:26.199041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.198996 4510 scope.go:122] "RemoveContainer" containerID="61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69" Nov 02 21:39:26.208946 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.208932 4510 scope.go:122] "RemoveContainer" containerID="61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69" Nov 02 21:39:26.209203 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:39:26.209185 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69\": container with ID starting with 61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69 not found: ID does not exist" containerID="61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69" Nov 02 21:39:26.209256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.209209 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69"} err="failed to get container status \"61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69\": rpc error: code = NotFound desc = could not find container \"61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69\": container with ID starting with 61bed4f46a477fc09bc3e04fa81e1c378d40ec9be57bef7c67d8720db77bca69 not found: ID does not exist" Nov 02 21:39:26.209256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.209224 4510 scope.go:122] "RemoveContainer" containerID="822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2" Nov 02 21:39:26.219638 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.219621 4510 scope.go:122] "RemoveContainer" containerID="822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2" Nov 02 21:39:26.219932 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:39:26.219906 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2\": container with ID starting with 822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2 not found: ID does not exist" containerID="822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2" Nov 02 21:39:26.220037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.219938 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2"} err="failed to get container status \"822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2\": rpc error: code = NotFound desc = could not find container \"822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2\": container with ID starting with 822a47163fe83e8d292a1209b506fb0ed93248c2ca4b40c11f76985e04e893e2 not found: ID does not exist" Nov 02 21:39:26.336678 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.336604 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx"] Nov 02 21:39:26.382585 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.382554 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-gvmnx"] Nov 02 21:39:26.483748 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.483716 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5"] Nov 02 21:39:26.487926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.487900 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tglx5"] Nov 02 21:39:26.587953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.587866 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw"] Nov 02 21:39:26.592142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.592118 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-zpmjw"] Nov 02 21:39:26.693289 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.693236 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb"] Nov 02 21:39:26.697268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:26.697245 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ef6b43-openshift-default-679658c459-jqdtb"] Nov 02 21:39:28.159773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:28.159734 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" path="/var/lib/kubelet/pods/17a339a5-da47-4fdc-9cf5-7d5ba07b735d/volumes" Nov 02 21:39:28.160437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:28.160415 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" path="/var/lib/kubelet/pods/5b866e72-25c8-4171-ad22-f5eff84395b7/volumes" Nov 02 21:39:28.161094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:28.161074 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" path="/var/lib/kubelet/pods/87732d55-1a01-4d5d-b699-19edbff59b35/volumes" Nov 02 21:39:28.162188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:28.162168 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" path="/var/lib/kubelet/pods/f26a9ed3-7950-4ad2-957d-0deb5beb99c0/volumes" Nov 02 21:39:34.088857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.088824 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5"] Nov 02 21:39:34.089704 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089684 4510 cpu_manager.go:414] "containerMap: removing container" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" Nov 02 21:39:34.089789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089706 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" Nov 02 21:39:34.089789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089745 4510 cpu_manager.go:414] "containerMap: removing container" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" Nov 02 21:39:34.089789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089751 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" Nov 02 21:39:34.089789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089760 4510 cpu_manager.go:414] "containerMap: removing container" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" Nov 02 21:39:34.089789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089765 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" Nov 02 21:39:34.089789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089780 4510 cpu_manager.go:414] "containerMap: removing container" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" Nov 02 21:39:34.089789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089785 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" Nov 02 21:39:34.090138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089946 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b866e72-25c8-4171-ad22-f5eff84395b7" containerName="istio-proxy" Nov 02 21:39:34.090138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089956 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a339a5-da47-4fdc-9cf5-7d5ba07b735d" containerName="istio-proxy" Nov 02 21:39:34.090138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089962 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="87732d55-1a01-4d5d-b699-19edbff59b35" containerName="istio-proxy" Nov 02 21:39:34.090138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.089975 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26a9ed3-7950-4ad2-957d-0deb5beb99c0" containerName="istio-proxy" Nov 02 21:39:34.095312 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.095292 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.176543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.165579 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad28c88-ba91-464a-8314-a9af24fa79b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.176543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.165658 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dft9x\" (UniqueName: \"kubernetes.io/projected/aad28c88-ba91-464a-8314-a9af24fa79b2-kube-api-access-dft9x\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.200688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.191974 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b983e02\"" Nov 02 21:39:34.200688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.192162 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b983e02-dockercfg-pjj2h\"" Nov 02 21:39:34.200688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.193187 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5"] Nov 02 21:39:34.253965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.253710 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9"] Nov 02 21:39:34.259496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.259472 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.267820 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.267788 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad28c88-ba91-464a-8314-a9af24fa79b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.267971 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.267844 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dft9x\" (UniqueName: \"kubernetes.io/projected/aad28c88-ba91-464a-8314-a9af24fa79b2-kube-api-access-dft9x\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.268076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.268060 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvvh\" (UniqueName: \"kubernetes.io/projected/eda81ded-2cad-4112-84db-1205b4024d03-kube-api-access-vcvvh\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-swpb9\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.268287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.268221 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda81ded-2cad-4112-84db-1205b4024d03-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-swpb9\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.268773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.268753 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad28c88-ba91-464a-8314-a9af24fa79b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.315853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.315815 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9"] Nov 02 21:39:34.350348 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.350260 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dft9x\" (UniqueName: \"kubernetes.io/projected/aad28c88-ba91-464a-8314-a9af24fa79b2-kube-api-access-dft9x\") pod \"payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.373979 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.371609 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvvh\" (UniqueName: \"kubernetes.io/projected/eda81ded-2cad-4112-84db-1205b4024d03-kube-api-access-vcvvh\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-swpb9\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.373979 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.371693 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda81ded-2cad-4112-84db-1205b4024d03-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-swpb9\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.373979 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.372677 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda81ded-2cad-4112-84db-1205b4024d03-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-swpb9\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.410470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.410436 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:34.426066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.426038 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvvh\" (UniqueName: \"kubernetes.io/projected/eda81ded-2cad-4112-84db-1205b4024d03-kube-api-access-vcvvh\") pod \"payload-processing-e2e-mt-6b983e02-66b998455c-swpb9\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.576651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.576485 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:34.802285 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.802254 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5"] Nov 02 21:39:34.808712 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:39:34.808684 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad28c88_ba91_464a_8314_a9af24fa79b2.slice/crio-de4fd9e04bfd1c44ff054129e841f1567f70b6694434a97402fcb83793c275b1 WatchSource:0}: Error finding container de4fd9e04bfd1c44ff054129e841f1567f70b6694434a97402fcb83793c275b1: Status 404 returned error can't find the container with id de4fd9e04bfd1c44ff054129e841f1567f70b6694434a97402fcb83793c275b1 Nov 02 21:39:34.966329 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:34.966241 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9"] Nov 02 21:39:35.265980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.265886 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" event={"ID":"eda81ded-2cad-4112-84db-1205b4024d03","Type":"ContainerStarted","Data":"efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10"} Nov 02 21:39:35.265980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.265943 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" event={"ID":"eda81ded-2cad-4112-84db-1205b4024d03","Type":"ContainerStarted","Data":"e546bc53d00100666b786f6eca1b417fbebe2c7e817b32f9138d48e8bff43cda"} Nov 02 21:39:35.266450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.266087 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:39:35.267571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.267548 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" event={"ID":"aad28c88-ba91-464a-8314-a9af24fa79b2","Type":"ContainerStarted","Data":"852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc"} Nov 02 21:39:35.267683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.267590 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:35.267683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.267610 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" event={"ID":"aad28c88-ba91-464a-8314-a9af24fa79b2","Type":"ContainerStarted","Data":"de4fd9e04bfd1c44ff054129e841f1567f70b6694434a97402fcb83793c275b1"} Nov 02 21:39:35.375692 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.375637 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" podStartSLOduration=1.375621193 podStartE2EDuration="1.375621193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:35.371313433 +0000 UTC m=+1601.347611208" watchObservedRunningTime="2026-11-02 21:39:35.375621193 +0000 UTC m=+1601.351918969" Nov 02 21:39:35.477146 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:35.477091 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" podStartSLOduration=1.477078384 podStartE2EDuration="1.477078384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:39:35.475265996 +0000 UTC m=+1601.451563772" watchObservedRunningTime="2026-11-02 21:39:35.477078384 +0000 UTC m=+1601.453376158" Nov 02 21:39:45.544971 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:39:45.544933 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:39:46.277485 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:46.277448 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:39:46.277659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:39:46.277559 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:40:07.769556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:07.769502 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq"] Nov 02 21:40:07.770060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:07.769724 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" podUID="3f335f26-4d8c-4fa2-b748-7a877375adb1" containerName="maas-api" containerID="cri-o://9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b" gracePeriod=30 Nov 02 21:40:08.057195 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.057160 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:40:08.151967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.151925 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-maas-api-tls\") pod \"3f335f26-4d8c-4fa2-b748-7a877375adb1\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " Nov 02 21:40:08.152166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.152018 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-metrics-certs\") pod \"3f335f26-4d8c-4fa2-b748-7a877375adb1\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " Nov 02 21:40:08.152402 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.152382 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f335f26-4d8c-4fa2-b748-7a877375adb1-kube-api-access-zgpcj\" (UniqueName: \"kubernetes.io/projected/3f335f26-4d8c-4fa2-b748-7a877375adb1-kube-api-access-zgpcj\") pod \"3f335f26-4d8c-4fa2-b748-7a877375adb1\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " Nov 02 21:40:08.152458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.152430 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f335f26-4d8c-4fa2-b748-7a877375adb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335f26-4d8c-4fa2-b748-7a877375adb1-openshift-service-ca\") pod \"3f335f26-4d8c-4fa2-b748-7a877375adb1\" (UID: \"3f335f26-4d8c-4fa2-b748-7a877375adb1\") " Nov 02 21:40:08.153498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.153463 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f335f26-4d8c-4fa2-b748-7a877375adb1-openshift-service-ca" pod "3f335f26-4d8c-4fa2-b748-7a877375adb1" (UID: "3f335f26-4d8c-4fa2-b748-7a877375adb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:08.159750 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.159717 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-maas-api-tls" pod "3f335f26-4d8c-4fa2-b748-7a877375adb1" (UID: "3f335f26-4d8c-4fa2-b748-7a877375adb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:08.159886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.159744 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-metrics-certs" pod "3f335f26-4d8c-4fa2-b748-7a877375adb1" (UID: "3f335f26-4d8c-4fa2-b748-7a877375adb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:08.160029 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.160004 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f335f26-4d8c-4fa2-b748-7a877375adb1-kube-api-access-zgpcj" pod "3f335f26-4d8c-4fa2-b748-7a877375adb1" (UID: "3f335f26-4d8c-4fa2-b748-7a877375adb1"). InnerVolumeSpecName "kube-api-access-zgpcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:08.255188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.255154 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:08.255188 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.255187 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f335f26-4d8c-4fa2-b748-7a877375adb1-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:08.255382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.255204 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgpcj\" (UniqueName: \"kubernetes.io/projected/3f335f26-4d8c-4fa2-b748-7a877375adb1-kube-api-access-zgpcj\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:08.255382 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.255221 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f335f26-4d8c-4fa2-b748-7a877375adb1-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:08.517421 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.517388 4510 generic.go:356] "Generic (PLEG): container finished" podID="3f335f26-4d8c-4fa2-b748-7a877375adb1" containerID="9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b" exitCode=0 Nov 02 21:40:08.517715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.517441 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" Nov 02 21:40:08.517715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.517462 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" event={"ID":"3f335f26-4d8c-4fa2-b748-7a877375adb1","Type":"ContainerDied","Data":"9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b"} Nov 02 21:40:08.517715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.517509 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq" event={"ID":"3f335f26-4d8c-4fa2-b748-7a877375adb1","Type":"ContainerDied","Data":"7317b6020fca86c64e9329d3bf86ba3d8289379f8ebc8dda5587d3cc8ccb9f23"} Nov 02 21:40:08.517715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.517554 4510 scope.go:122] "RemoveContainer" containerID="9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b" Nov 02 21:40:08.528779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.528755 4510 scope.go:122] "RemoveContainer" containerID="9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b" Nov 02 21:40:08.529058 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:08.529034 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b\": container with ID starting with 9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b not found: ID does not exist" containerID="9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b" Nov 02 21:40:08.529125 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.529069 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b"} err="failed to get container status \"9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b\": rpc error: code = NotFound desc = could not find container \"9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b\": container with ID starting with 9d5c90a2f69badde360f2cd8ac18875e5b5a5b4be11e8d3b6d667c0bf4b6315b not found: ID does not exist" Nov 02 21:40:08.671698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.671666 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq"] Nov 02 21:40:08.717298 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:08.717265 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7cd2d37-7989d88bf9-mslmq"] Nov 02 21:40:10.228943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:10.228907 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f335f26-4d8c-4fa2-b748-7a877375adb1" path="/var/lib/kubelet/pods/3f335f26-4d8c-4fa2-b748-7a877375adb1/volumes" Nov 02 21:40:15.531940 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:15.531899 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:40:24.978927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:24.978889 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc"] Nov 02 21:40:24.979650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:24.979634 4510 cpu_manager.go:414] "containerMap: removing container" podUID="3f335f26-4d8c-4fa2-b748-7a877375adb1" containerName="maas-api" Nov 02 21:40:24.979714 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:24.979652 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f335f26-4d8c-4fa2-b748-7a877375adb1" containerName="maas-api" Nov 02 21:40:24.979903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:24.979890 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f335f26-4d8c-4fa2-b748-7a877375adb1" containerName="maas-api" Nov 02 21:40:24.983775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:24.983754 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.027950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.027919 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7039890a-43f4-4d63-bc88-1a9d2c54f413-openshift-service-ca\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.028097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.027968 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.028097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.028070 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpzf\" (UniqueName: \"kubernetes.io/projected/7039890a-43f4-4d63-bc88-1a9d2c54f413-kube-api-access-9vpzf\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.028198 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.028128 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-metrics-certs\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.033099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.033061 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b983e02-serving-cert\"" Nov 02 21:40:25.045419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.045385 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc"] Nov 02 21:40:25.157546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.135239 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7039890a-43f4-4d63-bc88-1a9d2c54f413-openshift-service-ca\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.157546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.135302 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.157546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.135381 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpzf\" (UniqueName: \"kubernetes.io/projected/7039890a-43f4-4d63-bc88-1a9d2c54f413-kube-api-access-9vpzf\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.157546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.135469 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-metrics-certs\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.225882 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:25.159745 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-serving-cert: secret "maas-api-e2e-mt-6b983e02-serving-cert" not found Nov 02 21:40:25.225882 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:25.159832 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls podName:7039890a-43f4-4d63-bc88-1a9d2c54f413 nodeName:}" failed. No retries permitted until 2026-11-02 21:40:25.659811677 +0000 UTC m=+1651.636109443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls") pod "maas-api-e2e-mt-6b983e02-545d64b797-j88vc" (UID: "7039890a-43f4-4d63-bc88-1a9d2c54f413") : secret "maas-api-e2e-mt-6b983e02-serving-cert" not found Nov 02 21:40:25.225882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.160548 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7039890a-43f4-4d63-bc88-1a9d2c54f413-openshift-service-ca\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.267592 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.261303 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpzf\" (UniqueName: \"kubernetes.io/projected/7039890a-43f4-4d63-bc88-1a9d2c54f413-kube-api-access-9vpzf\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.301481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.301411 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-metrics-certs\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.672248 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.672208 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.674893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.674866 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-j88vc\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:25.899121 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:25.899088 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:26.290328 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:26.290241 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc"] Nov 02 21:40:26.666474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:26.666435 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" event={"ID":"7039890a-43f4-4d63-bc88-1a9d2c54f413","Type":"ContainerStarted","Data":"4a76196b0d5e26edec32083409d3aebc9b0f46965397bd682d49efbbfacf0772"} Nov 02 21:40:27.675592 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:27.675549 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" event={"ID":"7039890a-43f4-4d63-bc88-1a9d2c54f413","Type":"ContainerStarted","Data":"f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5"} Nov 02 21:40:27.676040 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:27.675754 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:27.782723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:27.782672 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" podStartSLOduration=2.801395008 podStartE2EDuration="3.782658609s" totalImagesPullingTime="981.263601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:27.782069306 +0000 UTC m=+1653.758367120" watchObservedRunningTime="2026-11-02 21:40:27.782658609 +0000 UTC m=+1653.758956383" Nov 02 21:40:30.260422 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.260361 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw"] Nov 02 21:40:30.264709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.264689 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.315643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.315616 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f4a242-serving-cert\"" Nov 02 21:40:30.354885 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.354851 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.355048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.355025 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.355372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.355348 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4233e721-fed9-4272-b083-700f52ffecda-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.355426 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.355405 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqnk9\" (UniqueName: \"kubernetes.io/projected/4233e721-fed9-4272-b083-700f52ffecda-kube-api-access-bqnk9\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.368095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.368071 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw"] Nov 02 21:40:30.457613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.457578 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4233e721-fed9-4272-b083-700f52ffecda-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.457843 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.457634 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqnk9\" (UniqueName: \"kubernetes.io/projected/4233e721-fed9-4272-b083-700f52ffecda-kube-api-access-bqnk9\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.457843 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.457677 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.457843 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.457805 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.458351 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.458324 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4233e721-fed9-4272-b083-700f52ffecda-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.458538 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:30.458421 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-serving-cert: secret "maas-api-e2e-shared-a-w2-f4a242-serving-cert" not found Nov 02 21:40:30.458538 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:30.458487 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls podName:4233e721-fed9-4272-b083-700f52ffecda nodeName:}" failed. No retries permitted until 2026-11-02 21:40:30.958468623 +0000 UTC m=+1656.934766381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls") pod "maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" (UID: "4233e721-fed9-4272-b083-700f52ffecda") : secret "maas-api-e2e-shared-a-w2-f4a242-serving-cert" not found Nov 02 21:40:30.460807 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.460783 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.511292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.511232 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqnk9\" (UniqueName: \"kubernetes.io/projected/4233e721-fed9-4272-b083-700f52ffecda-kube-api-access-bqnk9\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.968829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.968789 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:30.971234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:30.971211 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:31.178747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:31.178713 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:31.568421 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:31.568338 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw"] Nov 02 21:40:31.574080 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:40:31.574054 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4233e721_fed9_4272_b083_700f52ffecda.slice/crio-cbbc622d967f2ef0bba19c5a6f2d259190f13bef9eac0d5b41c824a4e510a017 WatchSource:0}: Error finding container cbbc622d967f2ef0bba19c5a6f2d259190f13bef9eac0d5b41c824a4e510a017: Status 404 returned error can't find the container with id cbbc622d967f2ef0bba19c5a6f2d259190f13bef9eac0d5b41c824a4e510a017 Nov 02 21:40:31.724472 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:31.724433 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" event={"ID":"4233e721-fed9-4272-b083-700f52ffecda","Type":"ContainerStarted","Data":"cbbc622d967f2ef0bba19c5a6f2d259190f13bef9eac0d5b41c824a4e510a017"} Nov 02 21:40:32.735436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:32.735347 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" event={"ID":"4233e721-fed9-4272-b083-700f52ffecda","Type":"ContainerStarted","Data":"1945dec15b030872ce7799a5d67f362c5cf560975181d5c94c152926e187025d"} Nov 02 21:40:32.735436 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:32.735422 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:32.843714 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:32.843664 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" podStartSLOduration=1.943553243 podStartE2EDuration="2.843649634s" totalImagesPullingTime="900.096391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:32.840703127 +0000 UTC m=+1658.817001127" watchObservedRunningTime="2026-11-02 21:40:32.843649634 +0000 UTC m=+1658.819947408" Nov 02 21:40:33.689164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:33.689137 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:40:36.533190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:36.533113 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm"] Nov 02 21:40:36.533647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:36.533359 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" containerID="cri-o://ab5a05dbffabdfec21aba619327af656dd8708c9f031aa115c524925e5c39b34" gracePeriod=30 Nov 02 21:40:36.579943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:36.579908 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz"] Nov 02 21:40:36.580150 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:36.580130 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" containerID="cri-o://59e5503ef564c727fab6a49705700280993e8b0458d4de733a0677dabb0a4abc" gracePeriod=30 Nov 02 21:40:36.584007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:36.583979 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n"] Nov 02 21:40:36.584205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:36.584183 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" containerID="cri-o://bdca68259a992fb38b93c31052a9f898a6f9b8b149c2b5a7c29c22a31738dddc" gracePeriod=30 Nov 02 21:40:38.022486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:38.022451 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:40:38.748098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:38.748069 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 21:40:40.545554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:40.545481 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:40:40.545983 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:40.545595 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:40:41.364898 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.364865 4510 patch_prober.go:29] interesting pod/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:40:41.365067 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.364916 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:40:41.826798 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.826765 4510 generic.go:356] "Generic (PLEG): container finished" podID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerID="59e5503ef564c727fab6a49705700280993e8b0458d4de733a0677dabb0a4abc" exitCode=0 Nov 02 21:40:41.827279 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.826992 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" event={"ID":"ee5271f4-91ba-4ddf-b35f-a1b06f503d32","Type":"ContainerDied","Data":"59e5503ef564c727fab6a49705700280993e8b0458d4de733a0677dabb0a4abc"} Nov 02 21:40:41.829229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.829111 4510 generic.go:356] "Generic (PLEG): container finished" podID="093783da-0908-4409-a259-ebdc628d6e07" containerID="ab5a05dbffabdfec21aba619327af656dd8708c9f031aa115c524925e5c39b34" exitCode=0 Nov 02 21:40:41.829229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.829154 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" event={"ID":"093783da-0908-4409-a259-ebdc628d6e07","Type":"ContainerDied","Data":"ab5a05dbffabdfec21aba619327af656dd8708c9f031aa115c524925e5c39b34"} Nov 02 21:40:41.831232 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.831208 4510 generic.go:356] "Generic (PLEG): container finished" podID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerID="bdca68259a992fb38b93c31052a9f898a6f9b8b149c2b5a7c29c22a31738dddc" exitCode=0 Nov 02 21:40:41.831375 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.831242 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" event={"ID":"ed3764eb-eac8-4784-8e81-f935b19bd206","Type":"ContainerDied","Data":"bdca68259a992fb38b93c31052a9f898a6f9b8b149c2b5a7c29c22a31738dddc"} Nov 02 21:40:41.902623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.902563 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:40:41.954917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.954891 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:40:41.958773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.958750 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:40:41.967945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.967923 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-certs\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.968028 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.967971 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-certs\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968082 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-podinfo\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.968147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968125 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-kube-api-access-qx556\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-kube-api-access-qx556\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968200 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968181 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-istio-token\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-istio-token\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968219 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3764eb-eac8-4784-8e81-f935b19bd206-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3764eb-eac8-4784-8e81-f935b19bd206-istiod-ca-cert\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.968251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968218 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-certs" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.968355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968253 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-certs" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.968355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968283 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-socket\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968346 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-token\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-token\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.968508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968379 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-envoy\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968428 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-data\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.968648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968508 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-kube-api-access-hxckp\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-kube-api-access-hxckp\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.968648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968559 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-socket" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.968648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968565 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/093783da-0908-4409-a259-ebdc628d6e07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093783da-0908-4409-a259-ebdc628d6e07-istio-podinfo\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968611 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3764eb-eac8-4784-8e81-f935b19bd206-istiod-ca-cert" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:41.968859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968668 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-credential-socket\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.968859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968697 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-data\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968724 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-credential-socket\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968756 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-data" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.968859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968767 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093783da-0908-4409-a259-ebdc628d6e07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093783da-0908-4409-a259-ebdc628d6e07-istiod-ca-cert\") pod \"093783da-0908-4409-a259-ebdc628d6e07\" (UID: \"093783da-0908-4409-a259-ebdc628d6e07\") " Nov 02 21:40:41.968859 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968809 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-socket\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.969170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968901 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-envoy\") pod \"ed3764eb-eac8-4784-8e81-f935b19bd206\" (UID: \"ed3764eb-eac8-4784-8e81-f935b19bd206\") " Nov 02 21:40:41.969170 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.968950 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-credential-socket" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.969670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.969645 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed3764eb-eac8-4784-8e81-f935b19bd206-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:41.969766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.969676 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:41.969766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.969693 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:41.969766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.969708 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:41.969766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.969723 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:41.969766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.969738 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:41.970972 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.970943 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-podinfo" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:41.971059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971040 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-istio-token" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:41.971104 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971075 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-kube-api-access-hxckp" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "kube-api-access-hxckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:41.971104 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971096 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-token" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:41.971283 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971207 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-credential-socket" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.971283 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971246 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-data" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.971459 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971289 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-kube-api-access-qx556" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "kube-api-access-qx556". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:41.971459 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971362 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-socket" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.971459 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971383 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093783da-0908-4409-a259-ebdc628d6e07-istiod-ca-cert" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:41.971459 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971419 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-envoy" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:41.971777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.971757 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/093783da-0908-4409-a259-ebdc628d6e07-istio-podinfo" pod "093783da-0908-4409-a259-ebdc628d6e07" (UID: "093783da-0908-4409-a259-ebdc628d6e07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:41.972453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:41.972437 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-envoy" pod "ed3764eb-eac8-4784-8e81-f935b19bd206" (UID: "ed3764eb-eac8-4784-8e81-f935b19bd206"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:42.070415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070382 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-token\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-token\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.070622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070446 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-certs\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.070622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070486 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-data\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.070622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070605 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-envoy\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.070779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070655 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istiod-ca-cert\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.070779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070677 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-podinfo\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.070779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070734 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-credential-socket\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.070779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070770 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-socket\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.071036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070856 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-kube-api-access-jb869\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-kube-api-access-jb869\") pod \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\" (UID: \"ee5271f4-91ba-4ddf-b35f-a1b06f503d32\") " Nov 02 21:40:42.071036 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.070965 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-data" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:42.071147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071036 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-certs" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:42.071214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071196 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-credential-socket" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:42.071275 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071249 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istiod-ca-cert" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:42.071491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071383 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-socket" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:42.071690 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071673 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxckp\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-kube-api-access-hxckp\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.071785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071696 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093783da-0908-4409-a259-ebdc628d6e07-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.071785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071710 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.071785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071724 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.071785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071739 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.071785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071755 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093783da-0908-4409-a259-ebdc628d6e07-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.071785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071768 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.071785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071782 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071796 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071809 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071826 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071840 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071854 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx556\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-kube-api-access-qx556\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071869 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093783da-0908-4409-a259-ebdc628d6e07-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071883 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed3764eb-eac8-4784-8e81-f935b19bd206-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071896 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093783da-0908-4409-a259-ebdc628d6e07-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.071909 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.072963 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.072934 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-token" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:42.073064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.073000 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-podinfo" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:40:42.073172 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.073146 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-envoy" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:40:42.073254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.073193 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-kube-api-access-jb869" pod "ee5271f4-91ba-4ddf-b35f-a1b06f503d32" (UID: "ee5271f4-91ba-4ddf-b35f-a1b06f503d32"). InnerVolumeSpecName "kube-api-access-jb869". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:42.173431 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.173313 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.173431 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.173332 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.173431 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.173341 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.173431 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.173351 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb869\" (UniqueName: \"kubernetes.io/projected/ee5271f4-91ba-4ddf-b35f-a1b06f503d32-kube-api-access-jb869\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:42.840223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.840138 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" Nov 02 21:40:42.840223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.840138 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm" event={"ID":"093783da-0908-4409-a259-ebdc628d6e07","Type":"ContainerDied","Data":"e8517cb8e26ae5317fc2ffcec355b68d527039313a05168f2d354159369e41d8"} Nov 02 21:40:42.840761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.840299 4510 scope.go:122] "RemoveContainer" containerID="ab5a05dbffabdfec21aba619327af656dd8708c9f031aa115c524925e5c39b34" Nov 02 21:40:42.842317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.842291 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" Nov 02 21:40:42.844469 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.844145 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n" event={"ID":"ed3764eb-eac8-4784-8e81-f935b19bd206","Type":"ContainerDied","Data":"33eface3593910b93740c9afa2410da4231e95631b84e94e523ccf7718fd53a5"} Nov 02 21:40:42.848594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.848568 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" event={"ID":"ee5271f4-91ba-4ddf-b35f-a1b06f503d32","Type":"ContainerDied","Data":"ff308a8a435293223c969fa375b97d46efcefecdd511ecd92982362c371804f6"} Nov 02 21:40:42.848707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.848653 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz" Nov 02 21:40:42.854440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.854422 4510 scope.go:122] "RemoveContainer" containerID="bdca68259a992fb38b93c31052a9f898a6f9b8b149c2b5a7c29c22a31738dddc" Nov 02 21:40:42.864171 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.864155 4510 scope.go:122] "RemoveContainer" containerID="59e5503ef564c727fab6a49705700280993e8b0458d4de733a0677dabb0a4abc" Nov 02 21:40:42.994565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:42.994535 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz"] Nov 02 21:40:43.041019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:43.040993 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-qrspz"] Nov 02 21:40:43.100120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:43.100044 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm"] Nov 02 21:40:43.106136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:43.106105 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-5khvm"] Nov 02 21:40:43.207441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:43.207409 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n"] Nov 02 21:40:43.211513 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:43.211488 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-lwg4n"] Nov 02 21:40:44.157118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:44.157084 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093783da-0908-4409-a259-ebdc628d6e07" path="/var/lib/kubelet/pods/093783da-0908-4409-a259-ebdc628d6e07/volumes" Nov 02 21:40:44.157823 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:44.157804 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" path="/var/lib/kubelet/pods/ed3764eb-eac8-4784-8e81-f935b19bd206/volumes" Nov 02 21:40:44.158442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:44.158426 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" path="/var/lib/kubelet/pods/ee5271f4-91ba-4ddf-b35f-a1b06f503d32/volumes" Nov 02 21:40:45.548125 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:45.548086 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:40:51.045380 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.045348 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq"] Nov 02 21:40:51.045826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.045621 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" podUID="bde135ac-2ebd-4377-a7fd-5aed73856855" containerName="maas-api" containerID="cri-o://d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4" gracePeriod=30 Nov 02 21:40:51.390745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.390717 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:40:51.519609 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.519571 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-metrics-certs\") pod \"bde135ac-2ebd-4377-a7fd-5aed73856855\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " Nov 02 21:40:51.519803 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.519625 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-maas-api-tls\") pod \"bde135ac-2ebd-4377-a7fd-5aed73856855\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " Nov 02 21:40:51.519803 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.519729 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde135ac-2ebd-4377-a7fd-5aed73856855-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde135ac-2ebd-4377-a7fd-5aed73856855-openshift-service-ca\") pod \"bde135ac-2ebd-4377-a7fd-5aed73856855\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " Nov 02 21:40:51.519938 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.519826 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde135ac-2ebd-4377-a7fd-5aed73856855-kube-api-access-b2cst\" (UniqueName: \"kubernetes.io/projected/bde135ac-2ebd-4377-a7fd-5aed73856855-kube-api-access-b2cst\") pod \"bde135ac-2ebd-4377-a7fd-5aed73856855\" (UID: \"bde135ac-2ebd-4377-a7fd-5aed73856855\") " Nov 02 21:40:51.521544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.521499 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde135ac-2ebd-4377-a7fd-5aed73856855-openshift-service-ca" pod "bde135ac-2ebd-4377-a7fd-5aed73856855" (UID: "bde135ac-2ebd-4377-a7fd-5aed73856855"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:40:51.523681 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.523656 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-metrics-certs" pod "bde135ac-2ebd-4377-a7fd-5aed73856855" (UID: "bde135ac-2ebd-4377-a7fd-5aed73856855"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:51.524590 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.524500 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-maas-api-tls" pod "bde135ac-2ebd-4377-a7fd-5aed73856855" (UID: "bde135ac-2ebd-4377-a7fd-5aed73856855"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:40:51.524675 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.524645 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde135ac-2ebd-4377-a7fd-5aed73856855-kube-api-access-b2cst" pod "bde135ac-2ebd-4377-a7fd-5aed73856855" (UID: "bde135ac-2ebd-4377-a7fd-5aed73856855"). InnerVolumeSpecName "kube-api-access-b2cst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:40:51.626390 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.626350 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:51.626390 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.626390 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde135ac-2ebd-4377-a7fd-5aed73856855-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:51.626654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.626408 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde135ac-2ebd-4377-a7fd-5aed73856855-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:51.626654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.626423 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2cst\" (UniqueName: \"kubernetes.io/projected/bde135ac-2ebd-4377-a7fd-5aed73856855-kube-api-access-b2cst\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:40:51.927621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.927535 4510 generic.go:356] "Generic (PLEG): container finished" podID="bde135ac-2ebd-4377-a7fd-5aed73856855" containerID="d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4" exitCode=0 Nov 02 21:40:51.927621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.927609 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" Nov 02 21:40:51.927842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.927607 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" event={"ID":"bde135ac-2ebd-4377-a7fd-5aed73856855","Type":"ContainerDied","Data":"d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4"} Nov 02 21:40:51.927842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.927731 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq" event={"ID":"bde135ac-2ebd-4377-a7fd-5aed73856855","Type":"ContainerDied","Data":"acf23147e642b4623b090cb837a0dd534c0c34e78757dd612ddd783b60562051"} Nov 02 21:40:51.927842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.927752 4510 scope.go:122] "RemoveContainer" containerID="d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4" Nov 02 21:40:51.939697 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.939672 4510 scope.go:122] "RemoveContainer" containerID="d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4" Nov 02 21:40:51.939951 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:40:51.939934 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4\": container with ID starting with d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4 not found: ID does not exist" containerID="d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4" Nov 02 21:40:51.940002 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:51.939960 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4"} err="failed to get container status \"d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4\": rpc error: code = NotFound desc = could not find container \"d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4\": container with ID starting with d046e5bbe4303b2eb652acb877c304a27f33a06773b8398bdded9e295d3bc7f4 not found: ID does not exist" Nov 02 21:40:52.081418 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:52.081372 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq"] Nov 02 21:40:52.086098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:52.086070 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e942516f-566c8d7d47-ppmpq"] Nov 02 21:40:52.158234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:52.158096 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde135ac-2ebd-4377-a7fd-5aed73856855" path="/var/lib/kubelet/pods/bde135ac-2ebd-4377-a7fd-5aed73856855/volumes" Nov 02 21:40:54.066243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.066211 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd"] Nov 02 21:40:54.066952 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.066938 4510 cpu_manager.go:414] "containerMap: removing container" podUID="bde135ac-2ebd-4377-a7fd-5aed73856855" containerName="maas-api" Nov 02 21:40:54.066996 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.066953 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde135ac-2ebd-4377-a7fd-5aed73856855" containerName="maas-api" Nov 02 21:40:54.066996 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.066966 4510 cpu_manager.go:414] "containerMap: removing container" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" Nov 02 21:40:54.066996 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.066971 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067010 4510 cpu_manager.go:414] "containerMap: removing container" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067015 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067030 4510 cpu_manager.go:414] "containerMap: removing container" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067035 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067125 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5271f4-91ba-4ddf-b35f-a1b06f503d32" containerName="istio-proxy" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067135 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="093783da-0908-4409-a259-ebdc628d6e07" containerName="istio-proxy" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067144 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde135ac-2ebd-4377-a7fd-5aed73856855" containerName="maas-api" Nov 02 21:40:54.067164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.067154 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3764eb-eac8-4784-8e81-f935b19bd206" containerName="istio-proxy" Nov 02 21:40:54.072025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.072006 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.133538 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.125796 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-9f77b7-openshift-default-dockercfg-pv94t\"" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301068 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istiod-ca-cert\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301116 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bld\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-kube-api-access-r5bld\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301200 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-credential-socket\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301234 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-token\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301298 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-data\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301323 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-podinfo\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301424 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-certs\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301674 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-envoy\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.304546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.301742 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-socket\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.314405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.314260 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd"] Nov 02 21:40:54.403622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.403586 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-data\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.403622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.403622 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-podinfo\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.403840 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.403720 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-certs\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.403934 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.403918 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-envoy\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404030 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404011 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-socket\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404111 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404030 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-data\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404202 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404121 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istiod-ca-cert\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404202 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404152 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bld\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-kube-api-access-r5bld\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404202 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404180 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-certs\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404253 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-credential-socket\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404280 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-socket\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404297 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-token\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404701 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404677 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-credential-socket\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.404955 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.404929 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istiod-ca-cert\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.406341 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.406317 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-envoy\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.406561 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.406503 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-podinfo\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.458770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.458737 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-token\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.458918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.458816 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bld\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-kube-api-access-r5bld\") pod \"e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:54.686026 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:54.685933 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:55.123535 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:55.123493 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd"] Nov 02 21:40:55.170356 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:40:55.170327 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6d30ce_9a52_400e_a7c3_0937edd9f564.slice/crio-60907db633bfc5bd34906fbfa55d7fd45c2dbbfa90e93baee4151556bbdbb8ce WatchSource:0}: Error finding container 60907db633bfc5bd34906fbfa55d7fd45c2dbbfa90e93baee4151556bbdbb8ce: Status 404 returned error can't find the container with id 60907db633bfc5bd34906fbfa55d7fd45c2dbbfa90e93baee4151556bbdbb8ce Nov 02 21:40:55.173184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:55.173142 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:55.173311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:55.173230 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:55.173311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:55.173273 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:40:56.029511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:56.029473 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" event={"ID":"aa6d30ce-9a52-400e-a7c3-0937edd9f564","Type":"ContainerStarted","Data":"88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a"} Nov 02 21:40:56.029680 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:56.029552 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" event={"ID":"aa6d30ce-9a52-400e-a7c3-0937edd9f564","Type":"ContainerStarted","Data":"60907db633bfc5bd34906fbfa55d7fd45c2dbbfa90e93baee4151556bbdbb8ce"} Nov 02 21:40:56.227016 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:56.226961 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podStartSLOduration=2.226946201 podStartE2EDuration="2.226946201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:40:56.18318939 +0000 UTC m=+1682.159487166" watchObservedRunningTime="2026-11-02 21:40:56.226946201 +0000 UTC m=+1682.203243975" Nov 02 21:40:56.256332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:56.256301 4510 scope.go:122] "RemoveContainer" containerID="ca02590464fa2805068da9bd7f82a7179031f1d754f3a0e1fc91efbf26a2ab90" Nov 02 21:40:56.686461 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:56.686430 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:56.692460 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:56.692437 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:57.121641 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:57.116028 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:40:57.133559 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:40:57.132145 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:41:06.172058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.172024 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc"] Nov 02 21:41:06.172646 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.172208 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" containerID="cri-o://615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70" gracePeriod=30 Nov 02 21:41:06.217320 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.217284 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn"] Nov 02 21:41:06.217571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.217540 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" containerID="cri-o://121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d" gracePeriod=30 Nov 02 21:41:06.223820 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.223788 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx"] Nov 02 21:41:06.224048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.224010 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" containerID="cri-o://db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9" gracePeriod=30 Nov 02 21:41:06.262732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.262707 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr"] Nov 02 21:41:06.262890 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.262869 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" containerID="cri-o://b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c" gracePeriod=30 Nov 02 21:41:06.476579 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.476462 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj"] Nov 02 21:41:06.476751 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:06.476705 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" containerID="cri-o://7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d" gracePeriod=30 Nov 02 21:41:08.918976 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:08.918941 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p"] Nov 02 21:41:08.927405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:08.926576 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:08.974993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:08.974959 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p"] Nov 02 21:41:09.031783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.031750 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-06366c-serving-cert\"" Nov 02 21:41:09.063849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.063805 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.064015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.063979 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lh7z\" (UniqueName: \"kubernetes.io/projected/4e4665b8-c27f-44ef-8114-4e692d3dd622-kube-api-access-9lh7z\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.064123 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.064098 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.064268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.064251 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4665b8-c27f-44ef-8114-4e692d3dd622-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.171729 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.171857 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lh7z\" (UniqueName: \"kubernetes.io/projected/4e4665b8-c27f-44ef-8114-4e692d3dd622-kube-api-access-9lh7z\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.171934 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.172078 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4665b8-c27f-44ef-8114-4e692d3dd622-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:09.172497 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-serving-cert: secret "maas-api-e2e-shared-a-w3-06366c-serving-cert" not found Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:09.172602 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls podName:4e4665b8-c27f-44ef-8114-4e692d3dd622 nodeName:}" failed. No retries permitted until 2026-11-02 21:41:09.672579862 +0000 UTC m=+1695.648877618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls") pod "maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" (UID: "4e4665b8-c27f-44ef-8114-4e692d3dd622") : secret "maas-api-e2e-shared-a-w3-06366c-serving-cert" not found Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.172889 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4665b8-c27f-44ef-8114-4e692d3dd622-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.173478 4510 patch_prober.go:29] interesting pod/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:41:09.173707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.173545 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:41:09.188868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.188654 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.228831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.228804 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lh7z\" (UniqueName: \"kubernetes.io/projected/4e4665b8-c27f-44ef-8114-4e692d3dd622-kube-api-access-9lh7z\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.680470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.680436 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.682920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.682899 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:09.848155 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:09.848124 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:10.230184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:10.230154 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p"] Nov 02 21:41:10.236208 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:41:10.236177 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4665b8_c27f_44ef_8114_4e692d3dd622.slice/crio-6324f74fdabcb2d576d61d592838de4e20e9c18aa4731d48b61c458e1c922558 WatchSource:0}: Error finding container 6324f74fdabcb2d576d61d592838de4e20e9c18aa4731d48b61c458e1c922558: Status 404 returned error can't find the container with id 6324f74fdabcb2d576d61d592838de4e20e9c18aa4731d48b61c458e1c922558 Nov 02 21:41:10.237711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:10.237690 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:41:11.223173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.223134 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" event={"ID":"4e4665b8-c27f-44ef-8114-4e692d3dd622","Type":"ContainerStarted","Data":"6324f74fdabcb2d576d61d592838de4e20e9c18aa4731d48b61c458e1c922558"} Nov 02 21:41:11.617917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.617884 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:41:11.675416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.675390 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:41:11.686207 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.686148 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:41:11.690668 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.690641 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:41:11.718187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718103 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-data\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718151 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-token\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-token\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718412 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718364 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-socket\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718470 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-podinfo\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718574 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d22544-4180-49fa-8819-27bd0e34cf6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d22544-4180-49fa-8819-27bd0e34cf6e-istiod-ca-cert\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718613 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-certs\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718673 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-kube-api-access-hlpdn\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-kube-api-access-hlpdn\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718780 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-envoy\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.718973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718793 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-data" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.718973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.718864 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-credential-socket\") pod \"56d22544-4180-49fa-8819-27bd0e34cf6e\" (UID: \"56d22544-4180-49fa-8819-27bd0e34cf6e\") " Nov 02 21:41:11.719732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.719547 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-socket" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.719850 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.719768 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-credential-socket" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.720158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.720054 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d22544-4180-49fa-8819-27bd0e34cf6e-istiod-ca-cert" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:11.720158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.720136 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-certs" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.720425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.720379 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.720425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.720403 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.721420 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.721392 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-envoy" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.721879 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.721844 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-kube-api-access-hlpdn" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "kube-api-access-hlpdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.722081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.722052 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-token" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.722969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.722945 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-podinfo" pod "56d22544-4180-49fa-8819-27bd0e34cf6e" (UID: "56d22544-4180-49fa-8819-27bd0e34cf6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:11.763503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.763483 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:41:11.821453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.821415 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-token\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-token\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.821637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.821465 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-data\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.821875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.821848 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-data" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.822062 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822040 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-socket\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.822146 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822119 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-certs\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.822206 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822156 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-envoy\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.822206 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822194 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fec9655-c924-4228-808e-78533a4c500d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fec9655-c924-4228-808e-78533a4c500d-istiod-ca-cert\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.822318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822231 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istiod-ca-cert\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.822318 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822278 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-credential-socket\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.822419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822354 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-kube-api-access-nmvxh\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-kube-api-access-nmvxh\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.822419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822403 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-credential-socket\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.822543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822448 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-kube-api-access-vk942\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-kube-api-access-vk942\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.822543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822483 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-envoy\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.822543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822512 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-podinfo\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.822718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822605 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istiod-ca-cert\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.822718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822644 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-istio-token\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-istio-token\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.822718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822673 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-certs\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.822718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822704 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istiod-ca-cert\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.822961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822752 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-certs\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.822961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822807 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-podinfo\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.822961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822805 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-credential-socket" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.822961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822821 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istiod-ca-cert" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:11.822961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822869 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-credential-socket\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.822961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.822910 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fec9655-c924-4228-808e-78533a4c500d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fec9655-c924-4228-808e-78533a4c500d-istio-podinfo\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823040 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-data\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823042 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-credential-socket" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823080 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-podinfo\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823106 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-envoy\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823154 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-socket\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823181 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-kube-api-access-7kdzd\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-kube-api-access-7kdzd\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823219 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-credential-socket\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.823273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823249 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-socket\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.823693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823327 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-socket\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.823693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823395 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-data\") pod \"9fec9655-c924-4228-808e-78533a4c500d\" (UID: \"9fec9655-c924-4228-808e-78533a4c500d\") " Nov 02 21:41:11.823693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823431 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-certs\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.823693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823495 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-data\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.823693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823567 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-token\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-token\") pod \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\" (UID: \"08d7eb5a-704b-46c5-91e1-3a50149d06cd\") " Nov 02 21:41:11.823693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823643 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-kube-api-access-b6v2h\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-kube-api-access-b6v2h\") pod \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\" (UID: \"7b68b66b-eb3b-4eff-8e07-c36076d211ee\") " Nov 02 21:41:11.823693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823674 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-token\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-token\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.824045 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.823731 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-envoy\") pod \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\" (UID: \"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3\") " Nov 02 21:41:11.824122 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824063 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-credential-socket" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.824281 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824261 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istiod-ca-cert" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824467 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-data" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824653 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-socket" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824698 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-certs" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824871 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlpdn\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-kube-api-access-hlpdn\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824892 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824905 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824919 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824932 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824946 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824961 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824974 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.824987 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825000 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825014 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825027 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d22544-4180-49fa-8819-27bd0e34cf6e-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825041 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825056 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d22544-4180-49fa-8819-27bd0e34cf6e-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825070 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825084 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d22544-4180-49fa-8819-27bd0e34cf6e-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825277 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-certs" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825330 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-credential-socket" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825746 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istiod-ca-cert" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:11.826007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825431 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-token" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.828500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.828472 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-certs" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.828730 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.828706 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-certs" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.828800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.828759 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-socket" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.829454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.829432 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fec9655-c924-4228-808e-78533a4c500d-istiod-ca-cert" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:41:11.829674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.829656 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-data" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.829760 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.825492 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-socket" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.829760 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.829720 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-envoy" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.831069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.830737 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-kube-api-access-vk942" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "kube-api-access-vk942". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.831069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.830810 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-istio-token" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.831069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.830864 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-podinfo" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:11.831069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.831004 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-socket" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.832084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.832055 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-data" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.832180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.832083 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-token" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.832180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.831992 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-kube-api-access-b6v2h" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "kube-api-access-b6v2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.832340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.832318 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-kube-api-access-nmvxh" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "kube-api-access-nmvxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.832447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.832429 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-podinfo" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:11.833179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.833153 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-envoy" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.833320 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.833298 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-envoy" pod "7b68b66b-eb3b-4eff-8e07-c36076d211ee" (UID: "7b68b66b-eb3b-4eff-8e07-c36076d211ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.833372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.833330 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-envoy" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:41:11.833610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.833578 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-token" pod "08d7eb5a-704b-46c5-91e1-3a50149d06cd" (UID: "08d7eb5a-704b-46c5-91e1-3a50149d06cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.833676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.833636 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-kube-api-access-7kdzd" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "kube-api-access-7kdzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:41:11.833791 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.833765 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fec9655-c924-4228-808e-78533a4c500d-istio-podinfo" pod "9fec9655-c924-4228-808e-78533a4c500d" (UID: "9fec9655-c924-4228-808e-78533a4c500d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:11.834958 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.834936 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-podinfo" pod "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" (UID: "0f6617d0-5ecb-4db5-bc87-d7d9f5611af3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:41:11.926553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926501 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926554 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926572 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6v2h\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-kube-api-access-b6v2h\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926588 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926601 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926614 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926626 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926640 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fec9655-c924-4228-808e-78533a4c500d-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926653 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvxh\" (UniqueName: \"kubernetes.io/projected/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-kube-api-access-nmvxh\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926669 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk942\" (UniqueName: \"kubernetes.io/projected/08d7eb5a-704b-46c5-91e1-3a50149d06cd-kube-api-access-vk942\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926683 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926696 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926709 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926724 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926739 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926753 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926767 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b68b66b-eb3b-4eff-8e07-c36076d211ee-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926780 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.926786 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926794 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fec9655-c924-4228-808e-78533a4c500d-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926808 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926821 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926836 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926850 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kdzd\" (UniqueName: \"kubernetes.io/projected/9fec9655-c924-4228-808e-78533a4c500d-kube-api-access-7kdzd\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926864 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08d7eb5a-704b-46c5-91e1-3a50149d06cd-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926878 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926891 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fec9655-c924-4228-808e-78533a4c500d-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:11.927754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:11.926903 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b68b66b-eb3b-4eff-8e07-c36076d211ee-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:41:12.231945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.231918 4510 generic.go:356] "Generic (PLEG): container finished" podID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerID="db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9" exitCode=0 Nov 02 21:41:12.232078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.231979 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" event={"ID":"7b68b66b-eb3b-4eff-8e07-c36076d211ee","Type":"ContainerDied","Data":"db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9"} Nov 02 21:41:12.232078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.232044 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" Nov 02 21:41:12.232299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.232271 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx" event={"ID":"7b68b66b-eb3b-4eff-8e07-c36076d211ee","Type":"ContainerDied","Data":"8c31f089fb1e42d424367bc0d90cb716bf7304f720f20f50dee6b72aa1078331"} Nov 02 21:41:12.232430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.232319 4510 scope.go:122] "RemoveContainer" containerID="db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9" Nov 02 21:41:12.233861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.233494 4510 generic.go:356] "Generic (PLEG): container finished" podID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerID="121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d" exitCode=0 Nov 02 21:41:12.233861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.233604 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" Nov 02 21:41:12.233861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.233638 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" event={"ID":"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3","Type":"ContainerDied","Data":"121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d"} Nov 02 21:41:12.233861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.233677 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn" event={"ID":"0f6617d0-5ecb-4db5-bc87-d7d9f5611af3","Type":"ContainerDied","Data":"8652ec1e7a8705fd867973c34801be2d41809dc5a801dc4fb6c1147036908c3a"} Nov 02 21:41:12.235540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.235459 4510 generic.go:356] "Generic (PLEG): container finished" podID="9fec9655-c924-4228-808e-78533a4c500d" containerID="b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c" exitCode=0 Nov 02 21:41:12.235678 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.235658 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" event={"ID":"9fec9655-c924-4228-808e-78533a4c500d","Type":"ContainerDied","Data":"b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c"} Nov 02 21:41:12.235747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.235706 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" event={"ID":"9fec9655-c924-4228-808e-78533a4c500d","Type":"ContainerDied","Data":"410805569b5cbb12102c4e84c106f400dfff209b4e6a94be1894b9dfc65664e3"} Nov 02 21:41:12.235810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.235776 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr" Nov 02 21:41:12.239069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.238899 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" event={"ID":"4e4665b8-c27f-44ef-8114-4e692d3dd622","Type":"ContainerStarted","Data":"344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85"} Nov 02 21:41:12.239069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.239038 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:12.240619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.240554 4510 generic.go:356] "Generic (PLEG): container finished" podID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerID="7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d" exitCode=0 Nov 02 21:41:12.240769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.240625 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" Nov 02 21:41:12.240769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.240658 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" event={"ID":"08d7eb5a-704b-46c5-91e1-3a50149d06cd","Type":"ContainerDied","Data":"7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d"} Nov 02 21:41:12.240769 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.240693 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj" event={"ID":"08d7eb5a-704b-46c5-91e1-3a50149d06cd","Type":"ContainerDied","Data":"7481f56846c150a7e417eb08d67644ff874628fe90982fb16ee6b3b6a716af93"} Nov 02 21:41:12.242549 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.242514 4510 generic.go:356] "Generic (PLEG): container finished" podID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerID="615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70" exitCode=0 Nov 02 21:41:12.242664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.242585 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" Nov 02 21:41:12.242664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.242613 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" event={"ID":"56d22544-4180-49fa-8819-27bd0e34cf6e","Type":"ContainerDied","Data":"615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70"} Nov 02 21:41:12.242664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.242647 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc" event={"ID":"56d22544-4180-49fa-8819-27bd0e34cf6e","Type":"ContainerDied","Data":"69f79451e606f7bdf578c706d0de5879f5559fc60bd5cb65f74361f283d269bf"} Nov 02 21:41:12.252401 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.252376 4510 scope.go:122] "RemoveContainer" containerID="7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42" Nov 02 21:41:12.310204 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.310179 4510 scope.go:122] "RemoveContainer" containerID="db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9" Nov 02 21:41:12.310684 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:12.310657 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9\": container with ID starting with db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9 not found: ID does not exist" containerID="db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9" Nov 02 21:41:12.310759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.310698 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9"} err="failed to get container status \"db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9\": rpc error: code = NotFound desc = could not find container \"db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9\": container with ID starting with db052a5e0ba390f03eea8157bd9eb473f8806e838f443bd42421eeb825f8bed9 not found: ID does not exist" Nov 02 21:41:12.310759 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.310723 4510 scope.go:122] "RemoveContainer" containerID="7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42" Nov 02 21:41:12.311077 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:12.311053 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42\": container with ID starting with 7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42 not found: ID does not exist" containerID="7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42" Nov 02 21:41:12.311147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.311088 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42"} err="failed to get container status \"7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42\": rpc error: code = NotFound desc = could not find container \"7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42\": container with ID starting with 7382a14c8ee6c17acaf263f84d79813082eba4ab2b85007ca27b5f7461d31f42 not found: ID does not exist" Nov 02 21:41:12.311147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.311112 4510 scope.go:122] "RemoveContainer" containerID="121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d" Nov 02 21:41:12.320821 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.320804 4510 scope.go:122] "RemoveContainer" containerID="121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d" Nov 02 21:41:12.321053 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:12.321030 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d\": container with ID starting with 121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d not found: ID does not exist" containerID="121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d" Nov 02 21:41:12.321099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.321058 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d"} err="failed to get container status \"121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d\": rpc error: code = NotFound desc = could not find container \"121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d\": container with ID starting with 121ba5b07913b0ce4f02165e12936752e428b3f2caf9fb99b44e97bc1015d65d not found: ID does not exist" Nov 02 21:41:12.321099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.321071 4510 scope.go:122] "RemoveContainer" containerID="b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c" Nov 02 21:41:12.330197 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.330181 4510 scope.go:122] "RemoveContainer" containerID="b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c" Nov 02 21:41:12.330495 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:12.330471 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c\": container with ID starting with b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c not found: ID does not exist" containerID="b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c" Nov 02 21:41:12.330620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.330497 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c"} err="failed to get container status \"b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c\": rpc error: code = NotFound desc = could not find container \"b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c\": container with ID starting with b8d74984d5bb15fa5d1f3467c477266446c1e65bc29228b94f85d28c7b99e80c not found: ID does not exist" Nov 02 21:41:12.330620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.330513 4510 scope.go:122] "RemoveContainer" containerID="7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d" Nov 02 21:41:12.339335 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.339317 4510 scope.go:122] "RemoveContainer" containerID="7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d" Nov 02 21:41:12.339614 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:12.339595 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d\": container with ID starting with 7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d not found: ID does not exist" containerID="7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d" Nov 02 21:41:12.339682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.339624 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d"} err="failed to get container status \"7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d\": rpc error: code = NotFound desc = could not find container \"7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d\": container with ID starting with 7900ca13f81811cffbe0c095b0d482703fa07056aa53808a4240c0ce6558a81d not found: ID does not exist" Nov 02 21:41:12.339682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.339640 4510 scope.go:122] "RemoveContainer" containerID="615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70" Nov 02 21:41:12.351057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.351039 4510 scope.go:122] "RemoveContainer" containerID="615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70" Nov 02 21:41:12.351382 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:12.351347 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70\": container with ID starting with 615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70 not found: ID does not exist" containerID="615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70" Nov 02 21:41:12.351607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.351576 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70"} err="failed to get container status \"615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70\": rpc error: code = NotFound desc = could not find container \"615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70\": container with ID starting with 615b3621df9cef7aa468bf2a831fe6044b08746c8a34d18803d1c697fc4feb70 not found: ID does not exist" Nov 02 21:41:12.352993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.352969 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx"] Nov 02 21:41:12.354813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.354792 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-4ttqx"] Nov 02 21:41:12.455056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.454957 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn"] Nov 02 21:41:12.458858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.458830 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-25ttn"] Nov 02 21:41:12.611923 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.611876 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" podStartSLOduration=3.621944193 podStartE2EDuration="4.611863767s" totalImagesPullingTime="989.919574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:12.609596058 +0000 UTC m=+1698.585893833" watchObservedRunningTime="2026-11-02 21:41:12.611863767 +0000 UTC m=+1698.588161542" Nov 02 21:41:12.719103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.718995 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr"] Nov 02 21:41:12.723874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.723846 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-285rr"] Nov 02 21:41:12.875428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.875396 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc"] Nov 02 21:41:12.880763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.880735 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e2b27-openshift-default-996dc4fdf-bmbxc"] Nov 02 21:41:12.978212 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.978095 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj"] Nov 02 21:41:12.982166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:12.982140 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-252cj"] Nov 02 21:41:14.179752 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:14.179715 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" path="/var/lib/kubelet/pods/08d7eb5a-704b-46c5-91e1-3a50149d06cd/volumes" Nov 02 21:41:14.180800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:14.180775 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" path="/var/lib/kubelet/pods/0f6617d0-5ecb-4db5-bc87-d7d9f5611af3/volumes" Nov 02 21:41:14.181319 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:14.181300 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" path="/var/lib/kubelet/pods/56d22544-4180-49fa-8819-27bd0e34cf6e/volumes" Nov 02 21:41:14.182384 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:14.182364 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" path="/var/lib/kubelet/pods/7b68b66b-eb3b-4eff-8e07-c36076d211ee/volumes" Nov 02 21:41:14.182962 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:14.182947 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fec9655-c924-4228-808e-78533a4c500d" path="/var/lib/kubelet/pods/9fec9655-c924-4228-808e-78533a4c500d/volumes" Nov 02 21:41:15.593543 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:15.590203 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:41:18.259131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:18.259097 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:41:28.272234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.272199 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz"] Nov 02 21:41:28.273067 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273048 4510 cpu_manager.go:414] "containerMap: removing container" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" Nov 02 21:41:28.273131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273068 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" Nov 02 21:41:28.273131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273092 4510 cpu_manager.go:414] "containerMap: removing container" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" Nov 02 21:41:28.273131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273098 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" Nov 02 21:41:28.273131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273108 4510 cpu_manager.go:414] "containerMap: removing container" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" Nov 02 21:41:28.273131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273113 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" Nov 02 21:41:28.273131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273122 4510 cpu_manager.go:414] "containerMap: removing container" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" Nov 02 21:41:28.273131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273127 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273146 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273151 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273169 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273174 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273284 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6617d0-5ecb-4db5-bc87-d7d9f5611af3" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273296 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fec9655-c924-4228-808e-78533a4c500d" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273303 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d7eb5a-704b-46c5-91e1-3a50149d06cd" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273309 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d22544-4180-49fa-8819-27bd0e34cf6e" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273318 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" Nov 02 21:41:28.273439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.273327 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b68b66b-eb3b-4eff-8e07-c36076d211ee" containerName="istio-proxy" Nov 02 21:41:28.277508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.277489 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.333608 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.333582 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz"] Nov 02 21:41:28.351142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.351115 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b7107c-serving-cert\"" Nov 02 21:41:28.385650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.385617 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.385798 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.385662 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-metrics-certs\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.385842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.385790 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62pw\" (UniqueName: \"kubernetes.io/projected/a62e2eb2-f380-4c14-8ddb-42a946d85202-kube-api-access-q62pw\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.385943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.385928 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e2eb2-f380-4c14-8ddb-42a946d85202-openshift-service-ca\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.487104 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.487071 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e2eb2-f380-4c14-8ddb-42a946d85202-openshift-service-ca\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.487267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.487213 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.487267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.487243 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-metrics-certs\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.487761 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:28.487359 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-serving-cert: secret "maas-api-e2e-api-a-b7107c-serving-cert" not found Nov 02 21:41:28.487761 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:28.487430 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls podName:a62e2eb2-f380-4c14-8ddb-42a946d85202 nodeName:}" failed. No retries permitted until 2026-11-02 21:41:28.987409277 +0000 UTC m=+1714.963707044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls") pod "maas-api-e2e-api-a-b7107c-67b687b789-r2whz" (UID: "a62e2eb2-f380-4c14-8ddb-42a946d85202") : secret "maas-api-e2e-api-a-b7107c-serving-cert" not found Nov 02 21:41:28.487761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.487458 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62pw\" (UniqueName: \"kubernetes.io/projected/a62e2eb2-f380-4c14-8ddb-42a946d85202-kube-api-access-q62pw\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.487912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.487890 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e2eb2-f380-4c14-8ddb-42a946d85202-openshift-service-ca\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.489856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.489831 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-metrics-certs\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.545210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.545127 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62pw\" (UniqueName: \"kubernetes.io/projected/a62e2eb2-f380-4c14-8ddb-42a946d85202-kube-api-access-q62pw\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.996062 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.996030 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:28.998403 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:28.998375 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls\") pod \"maas-api-e2e-api-a-b7107c-67b687b789-r2whz\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:29.192103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:29.192068 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:29.568931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:29.568846 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz"] Nov 02 21:41:29.574660 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:41:29.574634 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62e2eb2_f380_4c14_8ddb_42a946d85202.slice/crio-8625223a20a7cdb1c86165031a020ca317b3ad40f66dd89a52e2ad6ba02d15ad WatchSource:0}: Error finding container 8625223a20a7cdb1c86165031a020ca317b3ad40f66dd89a52e2ad6ba02d15ad: Status 404 returned error can't find the container with id 8625223a20a7cdb1c86165031a020ca317b3ad40f66dd89a52e2ad6ba02d15ad Nov 02 21:41:30.398438 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:30.398399 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" event={"ID":"a62e2eb2-f380-4c14-8ddb-42a946d85202","Type":"ContainerStarted","Data":"8625223a20a7cdb1c86165031a020ca317b3ad40f66dd89a52e2ad6ba02d15ad"} Nov 02 21:41:31.408064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:31.408026 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" event={"ID":"a62e2eb2-f380-4c14-8ddb-42a946d85202","Type":"ContainerStarted","Data":"c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3"} Nov 02 21:41:31.408467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:31.408091 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:31.529670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:31.525019 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" podStartSLOduration=2.645683534 podStartE2EDuration="3.525000197s" totalImagesPullingTime="879.316663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:41:31.520774701 +0000 UTC m=+1717.497072478" watchObservedRunningTime="2026-11-02 21:41:31.525000197 +0000 UTC m=+1717.501297971" Nov 02 21:41:37.420102 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:37.420071 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:41:45.536059 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:41:45.536018 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:41:52.911918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:41:52.911882 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:42:04.172483 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.172451 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w"] Nov 02 21:42:04.178948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.178928 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.227442 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.227399 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2fd9\" (UniqueName: \"kubernetes.io/projected/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-kube-api-access-w2fd9\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.227973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.227475 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.227973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.227600 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.227973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.227693 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.231452 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.231418 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w"] Nov 02 21:42:04.273013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.272968 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-9f77b7-serving-cert\"" Nov 02 21:42:04.329391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.329360 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2fd9\" (UniqueName: \"kubernetes.io/projected/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-kube-api-access-w2fd9\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.329632 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.329435 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.329632 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.329550 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.329632 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:42:04.329600 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-serving-cert: secret "maas-api-e2e-shared-a-w6-9f77b7-serving-cert" not found Nov 02 21:42:04.329837 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:42:04.329668 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls podName:dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2 nodeName:}" failed. No retries permitted until 2026-11-02 21:42:04.829647588 +0000 UTC m=+1750.805945358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls") pod "maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" (UID: "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2") : secret "maas-api-e2e-shared-a-w6-9f77b7-serving-cert" not found Nov 02 21:42:04.329837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.329690 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.330610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.330563 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.332757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.332733 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.383012 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.382984 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2fd9\" (UniqueName: \"kubernetes.io/projected/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-kube-api-access-w2fd9\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.837387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.837352 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:04.839847 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:04.839827 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:05.092893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:05.092807 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:05.473000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:05.472966 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w"] Nov 02 21:42:05.480296 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:42:05.480268 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa3f2a9_c39b_41f3_9e1f_7c8f0c38f2b2.slice/crio-189b80e0037955fc49aa89ae0a2ee85d5b4bbf18e7aa5af32ffd143cfae606eb WatchSource:0}: Error finding container 189b80e0037955fc49aa89ae0a2ee85d5b4bbf18e7aa5af32ffd143cfae606eb: Status 404 returned error can't find the container with id 189b80e0037955fc49aa89ae0a2ee85d5b4bbf18e7aa5af32ffd143cfae606eb Nov 02 21:42:05.647992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:05.647920 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" event={"ID":"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2","Type":"ContainerStarted","Data":"189b80e0037955fc49aa89ae0a2ee85d5b4bbf18e7aa5af32ffd143cfae606eb"} Nov 02 21:42:06.656113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:06.656034 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" event={"ID":"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2","Type":"ContainerStarted","Data":"d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da"} Nov 02 21:42:06.656476 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:06.656119 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:06.766709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:06.766647 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" podStartSLOduration=1.8794391799999999 podStartE2EDuration="2.766629827s" totalImagesPullingTime="887.190647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:42:06.762443237 +0000 UTC m=+1752.738741012" watchObservedRunningTime="2026-11-02 21:42:06.766629827 +0000 UTC m=+1752.742927606" Nov 02 21:42:12.669511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:12.669480 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:42:13.455438 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.455407 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp"] Nov 02 21:42:13.460043 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.460021 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.554719 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.554687 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-d4d045-serving-cert\"" Nov 02 21:42:13.560502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.560479 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp"] Nov 02 21:42:13.589304 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.589267 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.589464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.589344 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmr7c\" (UniqueName: \"kubernetes.io/projected/c74e302b-f55c-4158-a0d3-8a682e6c75eb-kube-api-access-jmr7c\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.589529 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.589468 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.589744 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.589718 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74e302b-f55c-4158-a0d3-8a682e6c75eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.691791 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.691758 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.692239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.691824 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmr7c\" (UniqueName: \"kubernetes.io/projected/c74e302b-f55c-4158-a0d3-8a682e6c75eb-kube-api-access-jmr7c\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.692239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.691951 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.692239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.692025 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74e302b-f55c-4158-a0d3-8a682e6c75eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.692239 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:42:13.692087 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-serving-cert: secret "maas-api-e2e-shared-a-w1-d4d045-serving-cert" not found Nov 02 21:42:13.692239 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:42:13.692164 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls podName:c74e302b-f55c-4158-a0d3-8a682e6c75eb nodeName:}" failed. No retries permitted until 2026-11-02 21:42:14.192144453 +0000 UTC m=+1760.168442228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls") pod "maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" (UID: "c74e302b-f55c-4158-a0d3-8a682e6c75eb") : secret "maas-api-e2e-shared-a-w1-d4d045-serving-cert" not found Nov 02 21:42:13.692713 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.692696 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74e302b-f55c-4158-a0d3-8a682e6c75eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.694139 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.694115 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:13.763800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:13.763719 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmr7c\" (UniqueName: \"kubernetes.io/projected/c74e302b-f55c-4158-a0d3-8a682e6c75eb-kube-api-access-jmr7c\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:14.198982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:14.198948 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:14.201609 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:14.201580 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:14.373848 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:14.373820 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:14.750762 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:14.750720 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp"] Nov 02 21:42:14.756659 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:42:14.756631 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74e302b_f55c_4158_a0d3_8a682e6c75eb.slice/crio-0eed6551e2ed8b35ca7f0ab5882a51de86f13987b3bc6ff36af7cfd8dab19d98 WatchSource:0}: Error finding container 0eed6551e2ed8b35ca7f0ab5882a51de86f13987b3bc6ff36af7cfd8dab19d98: Status 404 returned error can't find the container with id 0eed6551e2ed8b35ca7f0ab5882a51de86f13987b3bc6ff36af7cfd8dab19d98 Nov 02 21:42:15.534207 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:42:15.534169 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:42:15.743860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:15.743821 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" event={"ID":"c74e302b-f55c-4158-a0d3-8a682e6c75eb","Type":"ContainerStarted","Data":"54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845"} Nov 02 21:42:15.744096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:15.743887 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" event={"ID":"c74e302b-f55c-4158-a0d3-8a682e6c75eb","Type":"ContainerStarted","Data":"0eed6551e2ed8b35ca7f0ab5882a51de86f13987b3bc6ff36af7cfd8dab19d98"} Nov 02 21:42:15.744096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:15.743920 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:15.853629 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:15.853509 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" podStartSLOduration=1.982914203 podStartE2EDuration="2.853496789s" totalImagesPullingTime="870.582586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:42:15.851884848 +0000 UTC m=+1761.828182624" watchObservedRunningTime="2026-11-02 21:42:15.853496789 +0000 UTC m=+1761.829794563" Nov 02 21:42:21.756223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:21.756143 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:42:35.770350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:42:35.770313 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:42:45.541258 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:42:45.541213 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:43:00.730233 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.730194 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64"] Nov 02 21:43:00.735932 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.735906 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:00.790218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.790183 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64"] Nov 02 21:43:00.835180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.835144 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-32de96-dockercfg-psmv5\"" Nov 02 21:43:00.835702 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.835661 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c62a903-b11e-4011-8e2b-7d34022876c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:00.835843 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.835728 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skkvc\" (UniqueName: \"kubernetes.io/projected/7c62a903-b11e-4011-8e2b-7d34022876c5-kube-api-access-skkvc\") pod \"payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:00.836029 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.835990 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-32de96\"" Nov 02 21:43:00.888476 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.888444 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc"] Nov 02 21:43:00.893433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.893407 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:00.937088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.937057 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c62a903-b11e-4011-8e2b-7d34022876c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:00.937256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.937100 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skkvc\" (UniqueName: \"kubernetes.io/projected/7c62a903-b11e-4011-8e2b-7d34022876c5-kube-api-access-skkvc\") pod \"payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:00.937797 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.937763 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c62a903-b11e-4011-8e2b-7d34022876c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:00.946235 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.946209 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc"] Nov 02 21:43:00.991117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:00.991034 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skkvc\" (UniqueName: \"kubernetes.io/projected/7c62a903-b11e-4011-8e2b-7d34022876c5-kube-api-access-skkvc\") pod \"payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:01.038917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.038881 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcq4\" (UniqueName: \"kubernetes.io/projected/b096eae1-e575-472d-aba6-50da1218fb46-kube-api-access-6gcq4\") pod \"payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:01.039124 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.039081 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b096eae1-e575-472d-aba6-50da1218fb46-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:01.052185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.052149 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:01.150173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.140776 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcq4\" (UniqueName: \"kubernetes.io/projected/b096eae1-e575-472d-aba6-50da1218fb46-kube-api-access-6gcq4\") pod \"payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:01.150173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.140997 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b096eae1-e575-472d-aba6-50da1218fb46-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:01.150173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.141915 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b096eae1-e575-472d-aba6-50da1218fb46-plugins-config-volume\") pod \"payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:01.195794 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.194903 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcq4\" (UniqueName: \"kubernetes.io/projected/b096eae1-e575-472d-aba6-50da1218fb46-kube-api-access-6gcq4\") pod \"payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:01.210798 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.210779 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:01.450625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.450590 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64"] Nov 02 21:43:01.456797 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:43:01.456772 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c62a903_b11e_4011_8e2b_7d34022876c5.slice/crio-c7039c5a5acfe8138e0e9d7efafb8faf986be51ad79eb88be99d652fb398b114 WatchSource:0}: Error finding container c7039c5a5acfe8138e0e9d7efafb8faf986be51ad79eb88be99d652fb398b114: Status 404 returned error can't find the container with id c7039c5a5acfe8138e0e9d7efafb8faf986be51ad79eb88be99d652fb398b114 Nov 02 21:43:01.593509 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:01.593481 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc"] Nov 02 21:43:01.599581 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:43:01.599511 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096eae1_e575_472d_aba6_50da1218fb46.slice/crio-293050c79465819322b29269790878db3e004a6683cb2b69d4ef4e12a35654a3 WatchSource:0}: Error finding container 293050c79465819322b29269790878db3e004a6683cb2b69d4ef4e12a35654a3: Status 404 returned error can't find the container with id 293050c79465819322b29269790878db3e004a6683cb2b69d4ef4e12a35654a3 Nov 02 21:43:02.073999 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.073957 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" event={"ID":"7c62a903-b11e-4011-8e2b-7d34022876c5","Type":"ContainerStarted","Data":"c95840ab51ecf87d68ce55bb00d52b6d9c5c88f1bee5d69d863a0824a2e4d512"} Nov 02 21:43:02.074465 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.074026 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" event={"ID":"7c62a903-b11e-4011-8e2b-7d34022876c5","Type":"ContainerStarted","Data":"c7039c5a5acfe8138e0e9d7efafb8faf986be51ad79eb88be99d652fb398b114"} Nov 02 21:43:02.074465 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.074053 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:02.075487 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.075461 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" event={"ID":"b096eae1-e575-472d-aba6-50da1218fb46","Type":"ContainerStarted","Data":"79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707"} Nov 02 21:43:02.075610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.075507 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" event={"ID":"b096eae1-e575-472d-aba6-50da1218fb46","Type":"ContainerStarted","Data":"293050c79465819322b29269790878db3e004a6683cb2b69d4ef4e12a35654a3"} Nov 02 21:43:02.075610 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.075571 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:02.181857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.181796 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" podStartSLOduration=2.181772602 podStartE2EDuration="2.181772602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:02.179597123 +0000 UTC m=+1808.155894898" watchObservedRunningTime="2026-11-02 21:43:02.181772602 +0000 UTC m=+1808.158070375" Nov 02 21:43:02.288172 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.288103 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" podStartSLOduration=2.288086917 podStartE2EDuration="2.288086917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:02.284572973 +0000 UTC m=+1808.260870748" watchObservedRunningTime="2026-11-02 21:43:02.288086917 +0000 UTC m=+1808.264384693" Nov 02 21:43:02.374644 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.374610 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6"] Nov 02 21:43:02.379656 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.379635 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.430753 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.430721 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f22129-serving-cert\"" Nov 02 21:43:02.432865 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.432835 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6"] Nov 02 21:43:02.461706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.461663 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.461882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.461798 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.461977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.461899 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdj8\" (UniqueName: \"kubernetes.io/projected/5e57a579-8556-4966-afdd-ccf6f0e38c4e-kube-api-access-2jdj8\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.461977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.461927 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57a579-8556-4966-afdd-ccf6f0e38c4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.564052 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.564003 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdj8\" (UniqueName: \"kubernetes.io/projected/5e57a579-8556-4966-afdd-ccf6f0e38c4e-kube-api-access-2jdj8\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.564257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.564063 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57a579-8556-4966-afdd-ccf6f0e38c4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.564257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.564194 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.564346 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.564337 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.564595 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:43:02.564579 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-serving-cert: secret "maas-api-e2e-shared-b-w2-f22129-serving-cert" not found Nov 02 21:43:02.564656 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:43:02.564646 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls podName:5e57a579-8556-4966-afdd-ccf6f0e38c4e nodeName:}" failed. No retries permitted until 2026-11-02 21:43:03.064625 +0000 UTC m=+1809.040922767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" (UID: "5e57a579-8556-4966-afdd-ccf6f0e38c4e") : secret "maas-api-e2e-shared-b-w2-f22129-serving-cert" not found Nov 02 21:43:02.565214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.565187 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57a579-8556-4966-afdd-ccf6f0e38c4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.567105 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.567083 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:02.618126 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:02.618090 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdj8\" (UniqueName: \"kubernetes.io/projected/5e57a579-8556-4966-afdd-ccf6f0e38c4e-kube-api-access-2jdj8\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:03.079421 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:03.079384 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:03.082569 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:03.082537 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:03.294997 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:03.294964 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:03.677440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:03.677404 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6"] Nov 02 21:43:03.683870 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:43:03.683842 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e57a579_8556_4966_afdd_ccf6f0e38c4e.slice/crio-2af69215a468cc93ea6fae11c007ed89131a84c0cc524e877d3e34b4bb4fd155 WatchSource:0}: Error finding container 2af69215a468cc93ea6fae11c007ed89131a84c0cc524e877d3e34b4bb4fd155: Status 404 returned error can't find the container with id 2af69215a468cc93ea6fae11c007ed89131a84c0cc524e877d3e34b4bb4fd155 Nov 02 21:43:04.091147 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:04.091061 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" event={"ID":"5e57a579-8556-4966-afdd-ccf6f0e38c4e","Type":"ContainerStarted","Data":"2af69215a468cc93ea6fae11c007ed89131a84c0cc524e877d3e34b4bb4fd155"} Nov 02 21:43:05.101127 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:05.101043 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" event={"ID":"5e57a579-8556-4966-afdd-ccf6f0e38c4e","Type":"ContainerStarted","Data":"83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0"} Nov 02 21:43:05.101127 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:05.101114 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:05.211588 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:05.211532 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" podStartSLOduration=2.284299728 podStartE2EDuration="3.211501047s" totalImagesPullingTime="927.201319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:05.206809274 +0000 UTC m=+1811.183107073" watchObservedRunningTime="2026-11-02 21:43:05.211501047 +0000 UTC m=+1811.187798819" Nov 02 21:43:11.112278 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:11.112245 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 21:43:13.086432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:13.086399 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:43:13.086844 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:13.086591 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:43:15.531492 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:43:15.531459 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:43:18.717140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:18.717106 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:43:21.574276 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:21.574240 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps"] Nov 02 21:43:21.574736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:21.574509 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" containerID="cri-o://ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd" gracePeriod=30 Nov 02 21:43:24.277067 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:24.277032 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:43:24.277454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:24.277103 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:43:26.855782 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.855728 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.962809 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25ff849-68d8-495c-8e8a-78016c3c20cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25ff849-68d8-495c-8e8a-78016c3c20cd-istiod-ca-cert\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.962901 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-kube-api-access-292d4\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-kube-api-access-292d4\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.963003 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-envoy\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.963101 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-certs\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.963152 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-podinfo\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.963183 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-token\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-token\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.963310 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-credential-socket\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.963408 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-data\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.964439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.963459 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-socket\") pod \"a25ff849-68d8-495c-8e8a-78016c3c20cd\" (UID: \"a25ff849-68d8-495c-8e8a-78016c3c20cd\") " Nov 02 21:43:26.965224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.965197 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-socket" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:26.979112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.979076 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25ff849-68d8-495c-8e8a-78016c3c20cd-istiod-ca-cert" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:43:26.980365 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.979733 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-credential-socket" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:26.980365 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.979915 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-data" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:26.980365 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:26.980128 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-certs" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:27.009454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.009396 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-token" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:27.011393 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.011281 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-envoy" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:43:27.011553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.011432 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-podinfo" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:43:27.013693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.013656 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-kube-api-access-292d4" pod "a25ff849-68d8-495c-8e8a-78016c3c20cd" (UID: "a25ff849-68d8-495c-8e8a-78016c3c20cd"). InnerVolumeSpecName "kube-api-access-292d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:43:27.071130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071089 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071129 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071143 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25ff849-68d8-495c-8e8a-78016c3c20cd-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071159 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-292d4\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-kube-api-access-292d4\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071174 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071189 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071204 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071218 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25ff849-68d8-495c-8e8a-78016c3c20cd-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.071447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.071232 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25ff849-68d8-495c-8e8a-78016c3c20cd-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:43:27.278395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.278359 4510 generic.go:356] "Generic (PLEG): container finished" podID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerID="ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd" exitCode=0 Nov 02 21:43:27.278626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.278408 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" Nov 02 21:43:27.278626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.278446 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" event={"ID":"a25ff849-68d8-495c-8e8a-78016c3c20cd","Type":"ContainerDied","Data":"ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd"} Nov 02 21:43:27.278626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.278506 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps" event={"ID":"a25ff849-68d8-495c-8e8a-78016c3c20cd","Type":"ContainerDied","Data":"999d153a5ac5a062cba82f2750b898e72143c857200289f768435613745cfceb"} Nov 02 21:43:27.278626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.278544 4510 scope.go:122] "RemoveContainer" containerID="ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd" Nov 02 21:43:27.289359 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.289344 4510 scope.go:122] "RemoveContainer" containerID="ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd" Nov 02 21:43:27.289662 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:43:27.289644 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd\": container with ID starting with ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd not found: ID does not exist" containerID="ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd" Nov 02 21:43:27.289735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.289670 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd"} err="failed to get container status \"ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd\": rpc error: code = NotFound desc = could not find container \"ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd\": container with ID starting with ff5dc45248b4f05b8df1467ab8e0eea5a736edb27686c5e06d7951ecf151dbbd not found: ID does not exist" Nov 02 21:43:27.448542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.440004 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps"] Nov 02 21:43:27.448542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:27.448251 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-jltps"] Nov 02 21:43:28.161315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:28.161281 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" path="/var/lib/kubelet/pods/a25ff849-68d8-495c-8e8a-78016c3c20cd/volumes" Nov 02 21:43:37.362529 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:37.362493 4510 generic.go:356] "Generic (PLEG): container finished" podID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerID="88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a" exitCode=0 Nov 02 21:43:37.363022 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:37.362571 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" event={"ID":"aa6d30ce-9a52-400e-a7c3-0937edd9f564","Type":"ContainerDied","Data":"88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a"} Nov 02 21:43:37.363022 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:37.362959 4510 scope.go:122] "RemoveContainer" containerID="88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a" Nov 02 21:43:37.364616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:37.364586 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:43:37.364706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:37.364657 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:43:37.364706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:37.364687 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:43:38.371341 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:38.371300 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" event={"ID":"aa6d30ce-9a52-400e-a7c3-0937edd9f564","Type":"ContainerStarted","Data":"2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa"} Nov 02 21:43:38.686375 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:38.686283 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:43:38.687845 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:38.687818 4510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:43:38.687965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:38.687867 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 21:43:39.687288 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:39.687252 4510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:43:39.687703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:39.687317 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 21:43:40.686811 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:40.686778 4510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:43:40.687005 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:40.686837 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 21:43:41.687275 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:41.687235 4510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:43:41.687858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:41.687308 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 21:43:42.115999 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:42.115969 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:43:42.686968 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:42.686933 4510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 21:43:42.687136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:42.686995 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 21:43:43.703183 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:43.703149 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:43:43.704412 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:43.704383 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:43:45.554341 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:43:45.554294 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:43:57.677295 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.677259 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf"] Nov 02 21:43:57.678091 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.678072 4510 cpu_manager.go:414] "containerMap: removing container" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" Nov 02 21:43:57.678137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.678092 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" Nov 02 21:43:57.678239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.678230 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25ff849-68d8-495c-8e8a-78016c3c20cd" containerName="istio-proxy" Nov 02 21:43:57.683152 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.683132 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.776060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.776017 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f13b18-serving-cert\"" Nov 02 21:43:57.782509 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.782478 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf"] Nov 02 21:43:57.822284 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.822255 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c0c179-138d-476e-8aa0-43102e35f404-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.822514 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.822495 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.822615 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.822600 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.822667 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.822631 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2b9\" (UniqueName: \"kubernetes.io/projected/09c0c179-138d-476e-8aa0-43102e35f404-kube-api-access-js2b9\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.923897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.923863 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.924075 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.923942 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.924075 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.923981 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2b9\" (UniqueName: \"kubernetes.io/projected/09c0c179-138d-476e-8aa0-43102e35f404-kube-api-access-js2b9\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.924198 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.924088 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c0c179-138d-476e-8aa0-43102e35f404-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.924758 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.924734 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c0c179-138d-476e-8aa0-43102e35f404-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.926326 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.926302 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.926415 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.926340 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:57.977728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:57.977665 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2b9\" (UniqueName: \"kubernetes.io/projected/09c0c179-138d-476e-8aa0-43102e35f404-kube-api-access-js2b9\") pod \"maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:58.002010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:58.001974 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:58.378516 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:58.378482 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf"] Nov 02 21:43:58.384513 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:43:58.384489 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c0c179_138d_476e_8aa0_43102e35f404.slice/crio-81bbd84873d077c488f14fd23230d7c5ca4e9382097257e12a5cc27dba5fc882 WatchSource:0}: Error finding container 81bbd84873d077c488f14fd23230d7c5ca4e9382097257e12a5cc27dba5fc882: Status 404 returned error can't find the container with id 81bbd84873d077c488f14fd23230d7c5ca4e9382097257e12a5cc27dba5fc882 Nov 02 21:43:58.534822 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:58.534787 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" event={"ID":"09c0c179-138d-476e-8aa0-43102e35f404","Type":"ContainerStarted","Data":"81bbd84873d077c488f14fd23230d7c5ca4e9382097257e12a5cc27dba5fc882"} Nov 02 21:43:59.544740 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:59.544659 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" event={"ID":"09c0c179-138d-476e-8aa0-43102e35f404","Type":"ContainerStarted","Data":"2a8680e4debaec44f22ade5c483b4f34e72ed705bb38642abd55dca28591ec29"} Nov 02 21:43:59.545092 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:59.544863 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:43:59.655683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:43:59.655627 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" podStartSLOduration=1.796378696 podStartE2EDuration="2.655610455s" totalImagesPullingTime="859.231759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:43:59.654248589 +0000 UTC m=+1865.630546366" watchObservedRunningTime="2026-11-02 21:43:59.655610455 +0000 UTC m=+1865.631908232" Nov 02 21:44:05.555622 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:05.555589 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:44:15.532073 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:44:15.532035 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:44:22.072097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.072062 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj"] Nov 02 21:44:22.076633 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.076607 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.172084 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.172046 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-32de96-serving-cert\"" Nov 02 21:44:22.178471 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.178443 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj"] Nov 02 21:44:22.185935 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.185908 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ks27\" (UniqueName: \"kubernetes.io/projected/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-kube-api-access-2ks27\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.186048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.185966 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-maas-api-tls\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.186163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.186148 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-openshift-service-ca\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.186240 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.186227 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-metrics-certs\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.287317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.287286 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ks27\" (UniqueName: \"kubernetes.io/projected/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-kube-api-access-2ks27\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.287511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.287346 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-maas-api-tls\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.287511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.287497 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-openshift-service-ca\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.287689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.287616 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-metrics-certs\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.288277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.288253 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-openshift-service-ca\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.289920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.289898 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-maas-api-tls\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.290056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.290038 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-metrics-certs\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.339791 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.339736 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ks27\" (UniqueName: \"kubernetes.io/projected/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-kube-api-access-2ks27\") pod \"maas-api-e2e-api-b-32de96-664d5b767f-9zjdj\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.391482 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.391459 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:22.768697 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:22.768667 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj"] Nov 02 21:44:22.772931 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:44:22.772895 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140c5dd2_c6bb_433a_92b8_d0f2d2a57ec8.slice/crio-afe7f8d410e79ca455acb19ab9a7a9501f80cb731aeb585269299d8b675135a3 WatchSource:0}: Error finding container afe7f8d410e79ca455acb19ab9a7a9501f80cb731aeb585269299d8b675135a3: Status 404 returned error can't find the container with id afe7f8d410e79ca455acb19ab9a7a9501f80cb731aeb585269299d8b675135a3 Nov 02 21:44:23.713064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:23.713039 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" event={"ID":"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8","Type":"ContainerStarted","Data":"afe7f8d410e79ca455acb19ab9a7a9501f80cb731aeb585269299d8b675135a3"} Nov 02 21:44:24.721773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:24.721727 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" event={"ID":"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8","Type":"ContainerStarted","Data":"599b92039aaec497f1afff10d1f95b09bcbe10693fea7b870063bc583736b611"} Nov 02 21:44:24.722166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:24.721806 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:24.829111 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:24.829055 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" podStartSLOduration=1.943917743 podStartE2EDuration="2.829043594s" totalImagesPullingTime="885.125851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:24.82838916 +0000 UTC m=+1890.804686935" watchObservedRunningTime="2026-11-02 21:44:24.829043594 +0000 UTC m=+1890.805341368" Nov 02 21:44:30.733750 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:30.733722 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:44:31.682096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.682061 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm"] Nov 02 21:44:31.687706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.687678 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.754046 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.754005 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-metrics-certs\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.754636 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.754052 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.754636 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.754117 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpj2\" (UniqueName: \"kubernetes.io/projected/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-kube-api-access-8xpj2\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.754636 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.754198 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.782259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.782231 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a1bf7d-serving-cert\"" Nov 02 21:44:31.786455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.786426 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm"] Nov 02 21:44:31.855626 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.855590 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-metrics-certs\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.855858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.855827 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.856007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.855898 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpj2\" (UniqueName: \"kubernetes.io/projected/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-kube-api-access-8xpj2\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.856007 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.855974 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.856314 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:44:31.856294 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-serving-cert: secret "maas-api-e2e-api-b-a1bf7d-serving-cert" not found Nov 02 21:44:31.856497 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:44:31.856484 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls podName:8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1 nodeName:}" failed. No retries permitted until 2026-11-02 21:44:32.356459601 +0000 UTC m=+1898.332757373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls") pod "maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" (UID: "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1") : secret "maas-api-e2e-api-b-a1bf7d-serving-cert" not found Nov 02 21:44:31.856856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.856824 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-openshift-service-ca\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.858816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.858793 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-metrics-certs\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:31.911865 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:31.911844 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpj2\" (UniqueName: \"kubernetes.io/projected/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-kube-api-access-8xpj2\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:32.364344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:32.364310 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:32.366857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:32.366824 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls\") pod \"maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:32.603734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:32.603703 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:32.994667 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:32.994579 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm"] Nov 02 21:44:33.001080 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:44:33.001046 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bca1aa3_0f2d_4315_8ba7_5cbbf40ae6d1.slice/crio-e209c39ccca2d99d19f88c20fa984944256b56bce8fe18f0dd97ccf04df46311 WatchSource:0}: Error finding container e209c39ccca2d99d19f88c20fa984944256b56bce8fe18f0dd97ccf04df46311: Status 404 returned error can't find the container with id e209c39ccca2d99d19f88c20fa984944256b56bce8fe18f0dd97ccf04df46311 Nov 02 21:44:33.792315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:33.792274 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" event={"ID":"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1","Type":"ContainerStarted","Data":"e209c39ccca2d99d19f88c20fa984944256b56bce8fe18f0dd97ccf04df46311"} Nov 02 21:44:34.801613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:34.801576 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" event={"ID":"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1","Type":"ContainerStarted","Data":"ac2a42151c4664785afa1c675ee402a87a6484d84cf8ac4b728e0ef520ad3262"} Nov 02 21:44:34.802017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:34.801790 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:34.907597 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:34.907501 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" podStartSLOduration=3.083368082 podStartE2EDuration="3.907489057s" totalImagesPullingTime="824.120975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:44:34.90727199 +0000 UTC m=+1900.883569766" watchObservedRunningTime="2026-11-02 21:44:34.907489057 +0000 UTC m=+1900.883786831" Nov 02 21:44:38.550184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:38.550148 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9"] Nov 02 21:44:38.656466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:38.656431 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5"] Nov 02 21:44:40.814431 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:40.814403 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:44:43.550615 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:43.550568 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" podUID="eda81ded-2cad-4112-84db-1205b4024d03" containerName="payload-processing" containerID="cri-o://efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10" gracePeriod=25 Nov 02 21:44:43.659867 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:43.659788 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" podUID="aad28c88-ba91-464a-8314-a9af24fa79b2" containerName="payload-pre-processing" containerID="cri-o://852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc" gracePeriod=25 Nov 02 21:44:45.533179 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:44:45.533134 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:44:46.276098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:46.276055 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" podUID="eda81ded-2cad-4112-84db-1205b4024d03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:44:46.276359 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:46.276073 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" podUID="aad28c88-ba91-464a-8314-a9af24fa79b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:44:46.627694 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:46.627665 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:44:56.276327 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:56.276282 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" podUID="eda81ded-2cad-4112-84db-1205b4024d03" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:44:56.276327 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:44:56.276294 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" podUID="aad28c88-ba91-464a-8314-a9af24fa79b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:45:03.856853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:03.856823 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:45:03.957617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:03.957591 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:45:03.968754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:03.968691 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda81ded-2cad-4112-84db-1205b4024d03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda81ded-2cad-4112-84db-1205b4024d03-plugins-config-volume\") pod \"eda81ded-2cad-4112-84db-1205b4024d03\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " Nov 02 21:45:03.968914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:03.968860 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda81ded-2cad-4112-84db-1205b4024d03-kube-api-access-vcvvh\" (UniqueName: \"kubernetes.io/projected/eda81ded-2cad-4112-84db-1205b4024d03-kube-api-access-vcvvh\") pod \"eda81ded-2cad-4112-84db-1205b4024d03\" (UID: \"eda81ded-2cad-4112-84db-1205b4024d03\") " Nov 02 21:45:03.969880 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:03.969854 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda81ded-2cad-4112-84db-1205b4024d03-plugins-config-volume" pod "eda81ded-2cad-4112-84db-1205b4024d03" (UID: "eda81ded-2cad-4112-84db-1205b4024d03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:03.971672 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:03.971650 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda81ded-2cad-4112-84db-1205b4024d03-kube-api-access-vcvvh" pod "eda81ded-2cad-4112-84db-1205b4024d03" (UID: "eda81ded-2cad-4112-84db-1205b4024d03"). InnerVolumeSpecName "kube-api-access-vcvvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:04.017250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.017216 4510 generic.go:356] "Generic (PLEG): container finished" podID="eda81ded-2cad-4112-84db-1205b4024d03" containerID="efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10" exitCode=0 Nov 02 21:45:04.017425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.017267 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" Nov 02 21:45:04.017425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.017290 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" event={"ID":"eda81ded-2cad-4112-84db-1205b4024d03","Type":"ContainerDied","Data":"efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10"} Nov 02 21:45:04.017425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.017338 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9" event={"ID":"eda81ded-2cad-4112-84db-1205b4024d03","Type":"ContainerDied","Data":"e546bc53d00100666b786f6eca1b417fbebe2c7e817b32f9138d48e8bff43cda"} Nov 02 21:45:04.017425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.017374 4510 scope.go:122] "RemoveContainer" containerID="efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10" Nov 02 21:45:04.019011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.018988 4510 generic.go:356] "Generic (PLEG): container finished" podID="aad28c88-ba91-464a-8314-a9af24fa79b2" containerID="852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc" exitCode=0 Nov 02 21:45:04.019120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.019016 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" event={"ID":"aad28c88-ba91-464a-8314-a9af24fa79b2","Type":"ContainerDied","Data":"852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc"} Nov 02 21:45:04.019120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.019037 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" Nov 02 21:45:04.019120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.019061 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5" event={"ID":"aad28c88-ba91-464a-8314-a9af24fa79b2","Type":"ContainerDied","Data":"de4fd9e04bfd1c44ff054129e841f1567f70b6694434a97402fcb83793c275b1"} Nov 02 21:45:04.028854 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.028837 4510 scope.go:122] "RemoveContainer" containerID="efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10" Nov 02 21:45:04.029138 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:04.029121 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10\": container with ID starting with efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10 not found: ID does not exist" containerID="efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10" Nov 02 21:45:04.029179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.029149 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10"} err="failed to get container status \"efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10\": rpc error: code = NotFound desc = could not find container \"efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10\": container with ID starting with efb754af21e7ce77d53a3356ea2a6c33f366d16f4c33bfeaf6756d6f1669ff10 not found: ID does not exist" Nov 02 21:45:04.029179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.029167 4510 scope.go:122] "RemoveContainer" containerID="852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc" Nov 02 21:45:04.038399 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.038384 4510 scope.go:122] "RemoveContainer" containerID="852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc" Nov 02 21:45:04.038671 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:04.038653 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc\": container with ID starting with 852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc not found: ID does not exist" containerID="852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc" Nov 02 21:45:04.038730 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.038677 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc"} err="failed to get container status \"852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc\": rpc error: code = NotFound desc = could not find container \"852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc\": container with ID starting with 852d5bf6f980530fc6bad51f02fff17f638cea0eefc5f9ca37fa1607b42f98dc not found: ID does not exist" Nov 02 21:45:04.071129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.071016 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad28c88-ba91-464a-8314-a9af24fa79b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad28c88-ba91-464a-8314-a9af24fa79b2-plugins-config-volume\") pod \"aad28c88-ba91-464a-8314-a9af24fa79b2\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " Nov 02 21:45:04.071129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.071065 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad28c88-ba91-464a-8314-a9af24fa79b2-kube-api-access-dft9x\" (UniqueName: \"kubernetes.io/projected/aad28c88-ba91-464a-8314-a9af24fa79b2-kube-api-access-dft9x\") pod \"aad28c88-ba91-464a-8314-a9af24fa79b2\" (UID: \"aad28c88-ba91-464a-8314-a9af24fa79b2\") " Nov 02 21:45:04.072019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.071990 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad28c88-ba91-464a-8314-a9af24fa79b2-plugins-config-volume" pod "aad28c88-ba91-464a-8314-a9af24fa79b2" (UID: "aad28c88-ba91-464a-8314-a9af24fa79b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:04.072606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.072580 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcvvh\" (UniqueName: \"kubernetes.io/projected/eda81ded-2cad-4112-84db-1205b4024d03-kube-api-access-vcvvh\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:04.072752 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.072608 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad28c88-ba91-464a-8314-a9af24fa79b2-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:04.072752 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.072624 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eda81ded-2cad-4112-84db-1205b4024d03-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:04.074586 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.074565 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad28c88-ba91-464a-8314-a9af24fa79b2-kube-api-access-dft9x" pod "aad28c88-ba91-464a-8314-a9af24fa79b2" (UID: "aad28c88-ba91-464a-8314-a9af24fa79b2"). InnerVolumeSpecName "kube-api-access-dft9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:04.173664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.173636 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dft9x\" (UniqueName: \"kubernetes.io/projected/aad28c88-ba91-464a-8314-a9af24fa79b2-kube-api-access-dft9x\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:04.225732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.225699 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9"] Nov 02 21:45:04.233418 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.233387 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b983e02-66b998455c-swpb9"] Nov 02 21:45:04.427573 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.427534 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5"] Nov 02 21:45:04.434557 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:04.434499 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b983e02-58d76c55bc-jqfd5"] Nov 02 21:45:06.155949 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:06.155861 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad28c88-ba91-464a-8314-a9af24fa79b2" path="/var/lib/kubelet/pods/aad28c88-ba91-464a-8314-a9af24fa79b2/volumes" Nov 02 21:45:06.156570 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:06.156550 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda81ded-2cad-4112-84db-1205b4024d03" path="/var/lib/kubelet/pods/eda81ded-2cad-4112-84db-1205b4024d03/volumes" Nov 02 21:45:10.561498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.561463 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9"] Nov 02 21:45:10.562279 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.562263 4510 cpu_manager.go:414] "containerMap: removing container" podUID="eda81ded-2cad-4112-84db-1205b4024d03" containerName="payload-processing" Nov 02 21:45:10.562329 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.562281 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda81ded-2cad-4112-84db-1205b4024d03" containerName="payload-processing" Nov 02 21:45:10.562329 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.562298 4510 cpu_manager.go:414] "containerMap: removing container" podUID="aad28c88-ba91-464a-8314-a9af24fa79b2" containerName="payload-pre-processing" Nov 02 21:45:10.562329 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.562303 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad28c88-ba91-464a-8314-a9af24fa79b2" containerName="payload-pre-processing" Nov 02 21:45:10.562492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.562479 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda81ded-2cad-4112-84db-1205b4024d03" containerName="payload-processing" Nov 02 21:45:10.562561 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.562500 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad28c88-ba91-464a-8314-a9af24fa79b2" containerName="payload-pre-processing" Nov 02 21:45:10.567326 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.567308 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.615828 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.615800 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-5451a5-serving-cert\"" Nov 02 21:45:10.666614 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.666586 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9"] Nov 02 21:45:10.671180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.671152 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn827\" (UniqueName: \"kubernetes.io/projected/93e34ccd-1b03-4b2f-8a1f-3408627570da-kube-api-access-bn827\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.671308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.671210 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.671414 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.671391 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.671628 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.671607 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e34ccd-1b03-4b2f-8a1f-3408627570da-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.772995 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.772953 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e34ccd-1b03-4b2f-8a1f-3408627570da-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.773183 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.773146 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn827\" (UniqueName: \"kubernetes.io/projected/93e34ccd-1b03-4b2f-8a1f-3408627570da-kube-api-access-bn827\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.773286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.773259 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.773371 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.773356 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.773539 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:10.773514 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-serving-cert: secret "maas-api-e2e-shared-b-w6-5451a5-serving-cert" not found Nov 02 21:45:10.773608 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:10.773583 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls podName:93e34ccd-1b03-4b2f-8a1f-3408627570da nodeName:}" failed. No retries permitted until 2026-11-02 21:45:11.273566616 +0000 UTC m=+1937.249864383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls") pod "maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" (UID: "93e34ccd-1b03-4b2f-8a1f-3408627570da") : secret "maas-api-e2e-shared-b-w6-5451a5-serving-cert" not found Nov 02 21:45:10.773808 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.773787 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e34ccd-1b03-4b2f-8a1f-3408627570da-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.775880 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.775856 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:10.827797 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:10.827703 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn827\" (UniqueName: \"kubernetes.io/projected/93e34ccd-1b03-4b2f-8a1f-3408627570da-kube-api-access-bn827\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:11.280453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:11.280419 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:11.282912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:11.282887 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:11.482095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:11.482063 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:11.858866 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:11.858784 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9"] Nov 02 21:45:11.865025 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:45:11.864998 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e34ccd_1b03_4b2f_8a1f_3408627570da.slice/crio-50ae5bd0c1ecb00f9edb046dcc5db648ebac44ac68c7a6cf8a9c5176338b49fa WatchSource:0}: Error finding container 50ae5bd0c1ecb00f9edb046dcc5db648ebac44ac68c7a6cf8a9c5176338b49fa: Status 404 returned error can't find the container with id 50ae5bd0c1ecb00f9edb046dcc5db648ebac44ac68c7a6cf8a9c5176338b49fa Nov 02 21:45:12.078594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:12.078563 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" event={"ID":"93e34ccd-1b03-4b2f-8a1f-3408627570da","Type":"ContainerStarted","Data":"50ae5bd0c1ecb00f9edb046dcc5db648ebac44ac68c7a6cf8a9c5176338b49fa"} Nov 02 21:45:13.090747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:13.090649 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" event={"ID":"93e34ccd-1b03-4b2f-8a1f-3408627570da","Type":"ContainerStarted","Data":"e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445"} Nov 02 21:45:13.090747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:13.090704 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:13.200177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:13.200126 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" podStartSLOduration=2.342443517 podStartE2EDuration="3.200113016s" totalImagesPullingTime="857.669499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:13.196411336 +0000 UTC m=+1939.172709105" watchObservedRunningTime="2026-11-02 21:45:13.200113016 +0000 UTC m=+1939.176410845" Nov 02 21:45:15.544274 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:15.544239 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:45:19.104264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:19.104230 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:45:24.266058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.266022 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p"] Nov 02 21:45:24.271475 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.271447 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.319709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.319680 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-66c72e-serving-cert\"" Nov 02 21:45:24.370512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.370477 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p"] Nov 02 21:45:24.375015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.374987 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.375129 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.375091 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.375177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.375164 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5gxx\" (UniqueName: \"kubernetes.io/projected/a9801076-2ad0-4395-b7e0-a36d63fd4852-kube-api-access-r5gxx\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.375304 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.375291 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9801076-2ad0-4395-b7e0-a36d63fd4852-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.476933 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.476896 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.477091 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.477039 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.477091 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:24.477065 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-serving-cert: secret "maas-api-e2e-shared-b-w1-66c72e-serving-cert" not found Nov 02 21:45:24.477171 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.477102 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5gxx\" (UniqueName: \"kubernetes.io/projected/a9801076-2ad0-4395-b7e0-a36d63fd4852-kube-api-access-r5gxx\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.477171 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:24.477139 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls podName:a9801076-2ad0-4395-b7e0-a36d63fd4852 nodeName:}" failed. No retries permitted until 2026-11-02 21:45:24.977119657 +0000 UTC m=+1950.953417413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls") pod "maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" (UID: "a9801076-2ad0-4395-b7e0-a36d63fd4852") : secret "maas-api-e2e-shared-b-w1-66c72e-serving-cert" not found Nov 02 21:45:24.477277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.477259 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9801076-2ad0-4395-b7e0-a36d63fd4852-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.477894 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.477868 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9801076-2ad0-4395-b7e0-a36d63fd4852-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.479481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.479453 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.544548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.544450 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5gxx\" (UniqueName: \"kubernetes.io/projected/a9801076-2ad0-4395-b7e0-a36d63fd4852-kube-api-access-r5gxx\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.984245 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.984204 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:24.986785 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:24.986757 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:25.185705 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:25.185672 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:25.566286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:25.566204 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p"] Nov 02 21:45:25.572529 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:45:25.572483 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9801076_2ad0_4395_b7e0_a36d63fd4852.slice/crio-ed99c0f2d793e849863151d7d0896eab8a8a10a7c8d023238679ac945998bfcd WatchSource:0}: Error finding container ed99c0f2d793e849863151d7d0896eab8a8a10a7c8d023238679ac945998bfcd: Status 404 returned error can't find the container with id ed99c0f2d793e849863151d7d0896eab8a8a10a7c8d023238679ac945998bfcd Nov 02 21:45:26.194237 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:26.194202 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" event={"ID":"a9801076-2ad0-4395-b7e0-a36d63fd4852","Type":"ContainerStarted","Data":"ed99c0f2d793e849863151d7d0896eab8a8a10a7c8d023238679ac945998bfcd"} Nov 02 21:45:27.211139 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:27.211105 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" event={"ID":"a9801076-2ad0-4395-b7e0-a36d63fd4852","Type":"ContainerStarted","Data":"0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb"} Nov 02 21:45:27.211573 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:27.211228 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:27.328268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:27.328206 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" podStartSLOduration=2.40723097 podStartE2EDuration="3.328190087s" totalImagesPullingTime="920.959117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:27.317208598 +0000 UTC m=+1953.293506374" watchObservedRunningTime="2026-11-02 21:45:27.328190087 +0000 UTC m=+1953.304487863" Nov 02 21:45:33.223022 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:33.222995 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:45:35.531856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:35.531821 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:45:37.072831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:37.072795 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm"] Nov 02 21:45:37.073223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:37.072997 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" containerID="cri-o://7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc" gracePeriod=30 Nov 02 21:45:37.239946 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:37.239896 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh"] Nov 02 21:45:37.240906 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:37.240871 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" containerID="cri-o://e9f558c80f1156d3ae31a7f21d8c94861e3e87788760f6a33205761b2fbf0d51" gracePeriod=30 Nov 02 21:45:37.465225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:37.465181 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:45:37.465433 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:37.465259 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:45:38.486896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:38.486665 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7"] Nov 02 21:45:38.487743 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:38.487696 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" containerID="cri-o://08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9" gracePeriod=30 Nov 02 21:45:40.308584 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:40.308549 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:45:40.309093 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:40.308604 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:45:42.373625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.373601 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.438721 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-token\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-token\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.438772 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istiod-ca-cert\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.438935 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-kube-api-access-d8kcm\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-kube-api-access-d8kcm\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.438996 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-data\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.439069 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-credential-socket\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.439112 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-socket\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.439263 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-envoy\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.439352 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-certs\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.439500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.439394 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-podinfo\") pod \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\" (UID: \"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed\") " Nov 02 21:45:42.440737 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.440341 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-data" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.441213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.441009 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-credential-socket" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.441213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.441083 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.441814 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.441463 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-socket" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.441814 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.441640 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istiod-ca-cert" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:42.442069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.442039 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-certs" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.446032 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.445990 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-kube-api-access-d8kcm" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "kube-api-access-d8kcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:42.446244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.446212 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-podinfo" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:42.446345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.446303 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-token" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:42.446391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.446352 4510 generic.go:356] "Generic (PLEG): container finished" podID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerID="7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc" exitCode=0 Nov 02 21:45:42.446553 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.446539 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" Nov 02 21:45:42.446615 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.446594 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" event={"ID":"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed","Type":"ContainerDied","Data":"7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc"} Nov 02 21:45:42.446665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.446657 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm" event={"ID":"02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed","Type":"ContainerDied","Data":"4fdd1a8e5aa3e85adcd56ba39351c4d78dbf78fb8e71d47329451388a1001828"} Nov 02 21:45:42.446723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.446700 4510 scope.go:122] "RemoveContainer" containerID="7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc" Nov 02 21:45:42.448965 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.448943 4510 generic.go:356] "Generic (PLEG): container finished" podID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerID="e9f558c80f1156d3ae31a7f21d8c94861e3e87788760f6a33205761b2fbf0d51" exitCode=0 Nov 02 21:45:42.449117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.449084 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" event={"ID":"2e318998-65bd-40f8-be06-9e02a4a6b8aa","Type":"ContainerDied","Data":"e9f558c80f1156d3ae31a7f21d8c94861e3e87788760f6a33205761b2fbf0d51"} Nov 02 21:45:42.454819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.454794 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-envoy" pod "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" (UID: "02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.488674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.488652 4510 scope.go:122] "RemoveContainer" containerID="7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc" Nov 02 21:45:42.489035 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:42.489007 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc\": container with ID starting with 7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc not found: ID does not exist" containerID="7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc" Nov 02 21:45:42.489128 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.489041 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc"} err="failed to get container status \"7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc\": rpc error: code = NotFound desc = could not find container \"7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc\": container with ID starting with 7abec839bd3617a940a23a13ccc08902a223d1ef874ce2f2192a1bd2c90c7bcc not found: ID does not exist" Nov 02 21:45:42.499385 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.499363 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:45:42.542346 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542320 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-credential-socket\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.542495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542384 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-data\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.542495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542485 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-certs\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.542608 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542596 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-podinfo\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.542655 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542640 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-envoy\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.542711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542665 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-credential-socket" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.542768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542733 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-token\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-token\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.542825 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542770 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istiod-ca-cert\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.542825 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542797 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-certs" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.542940 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.542849 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-kube-api-access-z5j9b\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-kube-api-access-z5j9b\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.543042 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543018 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-socket\") pod \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\" (UID: \"2e318998-65bd-40f8-be06-9e02a4a6b8aa\") " Nov 02 21:45:42.543157 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543104 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istiod-ca-cert" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:42.543406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543379 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-data" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.543499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543414 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-socket" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.543836 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543815 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.543914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543840 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8kcm\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-kube-api-access-d8kcm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.543914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543856 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.543914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543870 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.543914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543885 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.543914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543900 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.543914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543914 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.544236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543930 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.544236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543944 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.544236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543958 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.544236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543972 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.544236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.543986 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.544236 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.544000 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.545001 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.544977 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-podinfo" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:42.545117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.545094 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-token" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:42.545733 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.545708 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-kube-api-access-z5j9b" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "kube-api-access-z5j9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:42.546242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.546218 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-envoy" pod "2e318998-65bd-40f8-be06-9e02a4a6b8aa" (UID: "2e318998-65bd-40f8-be06-9e02a4a6b8aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:42.646090 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.646059 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.646090 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.646092 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.646287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.646108 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.646287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.646123 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5j9b\" (UniqueName: \"kubernetes.io/projected/2e318998-65bd-40f8-be06-9e02a4a6b8aa-kube-api-access-z5j9b\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:42.861702 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.861647 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm"] Nov 02 21:45:42.867851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:42.867821 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-48wgm"] Nov 02 21:45:43.458337 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:43.458312 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" Nov 02 21:45:43.458816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:43.458347 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh" event={"ID":"2e318998-65bd-40f8-be06-9e02a4a6b8aa","Type":"ContainerDied","Data":"86696cb682780abb341788be0e76d5865eb4171b76994ca464402af4c70f6b46"} Nov 02 21:45:43.458816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:43.458407 4510 scope.go:122] "RemoveContainer" containerID="e9f558c80f1156d3ae31a7f21d8c94861e3e87788760f6a33205761b2fbf0d51" Nov 02 21:45:43.614079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:43.614041 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh"] Nov 02 21:45:43.617791 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:43.617754 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-7twrh"] Nov 02 21:45:43.860973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:43.860893 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 02 21:45:43.860973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:43.860959 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 02 21:45:44.162651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.162615 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" path="/var/lib/kubelet/pods/02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed/volumes" Nov 02 21:45:44.163695 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.163668 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" path="/var/lib/kubelet/pods/2e318998-65bd-40f8-be06-9e02a4a6b8aa/volumes" Nov 02 21:45:44.216627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.216588 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:45:44.268708 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.268672 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-socket\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.268875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.268732 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-credential-socket\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.268875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.268777 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-envoy\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.268875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.268816 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-kube-api-access-q7jvn\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-kube-api-access-q7jvn\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.268875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.268862 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-data\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.269069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.268925 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a8f8047-52a3-408d-85f1-554d42416e76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8f8047-52a3-408d-85f1-554d42416e76-istio-podinfo\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.269069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.268947 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-socket" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.269069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.269007 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-istio-token\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-istio-token\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.269179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.269092 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-certs\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.269179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.269147 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8f8047-52a3-408d-85f1-554d42416e76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8f8047-52a3-408d-85f1-554d42416e76-istiod-ca-cert\") pod \"9a8f8047-52a3-408d-85f1-554d42416e76\" (UID: \"9a8f8047-52a3-408d-85f1-554d42416e76\") " Nov 02 21:45:44.274114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.270108 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.274114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.270644 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-data" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.274114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.270793 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-certs" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.274114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.272766 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a8f8047-52a3-408d-85f1-554d42416e76-istio-podinfo" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:44.274114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.272891 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-istio-token" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:44.274114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.274080 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8f8047-52a3-408d-85f1-554d42416e76-istiod-ca-cert" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:44.274748 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.274702 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-kube-api-access-q7jvn" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "kube-api-access-q7jvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:44.275120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.274997 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-credential-socket" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.286148 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.286113 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-envoy" pod "9a8f8047-52a3-408d-85f1-554d42416e76" (UID: "9a8f8047-52a3-408d-85f1-554d42416e76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:44.371702 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371670 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.371702 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371700 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.371945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371714 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7jvn\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-kube-api-access-q7jvn\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.371945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371727 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.371945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371739 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8f8047-52a3-408d-85f1-554d42416e76-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.371945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371751 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8f8047-52a3-408d-85f1-554d42416e76-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.371945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371764 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8f8047-52a3-408d-85f1-554d42416e76-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.371945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.371777 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8f8047-52a3-408d-85f1-554d42416e76-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:44.467722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.467627 4510 generic.go:356] "Generic (PLEG): container finished" podID="9a8f8047-52a3-408d-85f1-554d42416e76" containerID="08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9" exitCode=0 Nov 02 21:45:44.467722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.467680 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" Nov 02 21:45:44.467722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.467709 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" event={"ID":"9a8f8047-52a3-408d-85f1-554d42416e76","Type":"ContainerDied","Data":"08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9"} Nov 02 21:45:44.468244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.467763 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7" event={"ID":"9a8f8047-52a3-408d-85f1-554d42416e76","Type":"ContainerDied","Data":"e980438e625c460317ce69a1ce29b0eba445c877240d9166c432ac0a6fe060fd"} Nov 02 21:45:44.468244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.467784 4510 scope.go:122] "RemoveContainer" containerID="08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9" Nov 02 21:45:44.479644 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.479625 4510 scope.go:122] "RemoveContainer" containerID="08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9" Nov 02 21:45:44.479903 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:44.479882 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9\": container with ID starting with 08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9 not found: ID does not exist" containerID="08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9" Nov 02 21:45:44.479953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.479912 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9"} err="failed to get container status \"08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9\": rpc error: code = NotFound desc = could not find container \"08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9\": container with ID starting with 08a131150729c35dc54f3c1649a40c220efb9edf9801f6d1af3b215e41b03bb9 not found: ID does not exist" Nov 02 21:45:44.628250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.628216 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7"] Nov 02 21:45:44.631846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:44.631822 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-gh4d7"] Nov 02 21:45:45.533846 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:45:45.533810 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:45:46.156213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:46.156177 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" path="/var/lib/kubelet/pods/9a8f8047-52a3-408d-85f1-554d42416e76/volumes" Nov 02 21:45:46.212809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:46.212781 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k"] Nov 02 21:45:46.212989 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:46.212970 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" containerID="cri-o://f34cc388568f418d2e72e068439e1f6a9bbec26244ec96a35b22b2edd8ff65e9" gracePeriod=30 Nov 02 21:45:46.259086 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:46.259050 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh"] Nov 02 21:45:46.259300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:46.259262 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" containerID="cri-o://8f35793f12da742476be6263f2e0269b9ef00f9d6b79fafa4f7e53b8e4fadf50" gracePeriod=30 Nov 02 21:45:49.333648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.333606 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9"] Nov 02 21:45:49.334450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334432 4510 cpu_manager.go:414] "containerMap: removing container" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" Nov 02 21:45:49.334546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334451 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" Nov 02 21:45:49.334546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334534 4510 cpu_manager.go:414] "containerMap: removing container" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" Nov 02 21:45:49.334546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334541 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" Nov 02 21:45:49.334732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334570 4510 cpu_manager.go:414] "containerMap: removing container" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" Nov 02 21:45:49.334732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334578 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" Nov 02 21:45:49.334732 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334729 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e318998-65bd-40f8-be06-9e02a4a6b8aa" containerName="istio-proxy" Nov 02 21:45:49.334829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334740 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8f8047-52a3-408d-85f1-554d42416e76" containerName="istio-proxy" Nov 02 21:45:49.334829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.334750 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="02178fd7-d0a9-4ee0-ad2c-fa98bc6ddeed" containerName="istio-proxy" Nov 02 21:45:49.338500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.338483 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.389624 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.389606 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-5451a5-openshift-default-dockercfg-7jfx2\"" Nov 02 21:45:49.450659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.450616 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.450892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.450867 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.451009 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.450929 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.451073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.451007 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.451073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.451024 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlbm\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-kube-api-access-cwlbm\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.451178 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.451086 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3f2691-eb8c-413b-b17c-5938212aeec1-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.451178 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.451132 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.451282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.451199 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.451282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.451223 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.530906 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.530865 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9"] Nov 02 21:45:49.553115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553066 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.553115 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553113 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.553404 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553190 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.553404 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553218 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlbm\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-kube-api-access-cwlbm\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.553404 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553302 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3f2691-eb8c-413b-b17c-5938212aeec1-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.553404 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553356 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553421 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553502 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.553549 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.554053 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.554105 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.554301 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.554623 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.554831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.554807 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3f2691-eb8c-413b-b17c-5938212aeec1-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.556667 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.556644 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.556833 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.556812 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.607625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.607556 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.607742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.607709 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlbm\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-kube-api-access-cwlbm\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:49.653299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:49.653270 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:50.086887 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.086853 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9"] Nov 02 21:45:50.133696 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:45:50.133666 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3f2691_eb8c_413b_b17c_5938212aeec1.slice/crio-64827458ecdc16564a1f9b0eb06acca2fe8fb2576b4ba46d78139890f0d073ed WatchSource:0}: Error finding container 64827458ecdc16564a1f9b0eb06acca2fe8fb2576b4ba46d78139890f0d073ed: Status 404 returned error can't find the container with id 64827458ecdc16564a1f9b0eb06acca2fe8fb2576b4ba46d78139890f0d073ed Nov 02 21:45:50.136540 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.136494 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:50.136630 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.136578 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:50.136630 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.136611 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:45:50.515672 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.515638 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" event={"ID":"cf3f2691-eb8c-413b-b17c-5938212aeec1","Type":"ContainerStarted","Data":"53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87"} Nov 02 21:45:50.516172 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.515695 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" event={"ID":"cf3f2691-eb8c-413b-b17c-5938212aeec1","Type":"ContainerStarted","Data":"64827458ecdc16564a1f9b0eb06acca2fe8fb2576b4ba46d78139890f0d073ed"} Nov 02 21:45:50.624946 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.624871 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podStartSLOduration=1.624851741 podStartE2EDuration="1.624851741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:45:50.623820842 +0000 UTC m=+1976.600118617" watchObservedRunningTime="2026-11-02 21:45:50.624851741 +0000 UTC m=+1976.601149517" Nov 02 21:45:50.654435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:50.654395 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:51.538480 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.538272 4510 generic.go:356] "Generic (PLEG): container finished" podID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerID="f34cc388568f418d2e72e068439e1f6a9bbec26244ec96a35b22b2edd8ff65e9" exitCode=0 Nov 02 21:45:51.538480 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.538401 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" event={"ID":"0c9632c7-52f5-4223-aeb4-4b2fcef13522","Type":"ContainerDied","Data":"f34cc388568f418d2e72e068439e1f6a9bbec26244ec96a35b22b2edd8ff65e9"} Nov 02 21:45:51.541985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.540754 4510 generic.go:356] "Generic (PLEG): container finished" podID="1c508601-c5f7-440f-b605-fb035d963673" containerID="8f35793f12da742476be6263f2e0269b9ef00f9d6b79fafa4f7e53b8e4fadf50" exitCode=0 Nov 02 21:45:51.541985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.541920 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" event={"ID":"1c508601-c5f7-440f-b605-fb035d963673","Type":"ContainerDied","Data":"8f35793f12da742476be6263f2e0269b9ef00f9d6b79fafa4f7e53b8e4fadf50"} Nov 02 21:45:51.662541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.655197 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:45:51.662541 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.655253 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 21:45:51.662772 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.662751 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:45:51.700953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.700931 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:45:51.789956 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.789868 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c508601-c5f7-440f-b605-fb035d963673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c508601-c5f7-440f-b605-fb035d963673-istiod-ca-cert\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.789956 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.789911 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-envoy\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.789966 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-podinfo\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790016 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-socket\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.790186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790079 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-token\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-token\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790122 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istiod-ca-cert\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790156 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-kube-api-access-9l27f\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-kube-api-access-9l27f\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790214 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-envoy\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.790481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790247 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-kube-api-access-tjj8p\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-kube-api-access-tjj8p\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.790481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790268 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c508601-c5f7-440f-b605-fb035d963673-istiod-ca-cert" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:51.790481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790306 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-credential-socket\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.790481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790361 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-certs\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790373 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-socket" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.790481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790385 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-data\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790486 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c508601-c5f7-440f-b605-fb035d963673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c508601-c5f7-440f-b605-fb035d963673-istio-podinfo\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.790909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790535 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-certs\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.790909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790566 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-credential-socket\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.790909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790644 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-credential-socket" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.791130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.790959 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-certs" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.791130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.791015 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-data" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.791283 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.791260 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istiod-ca-cert" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:45:51.791534 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.791501 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-credential-socket" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.791810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.791787 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-socket\") pod \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\" (UID: \"0c9632c7-52f5-4223-aeb4-4b2fcef13522\") " Nov 02 21:45:51.791899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.791833 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-certs" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.792044 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-socket" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.791861 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-istio-token\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-istio-token\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.792205 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-data\") pod \"1c508601-c5f7-440f-b605-fb035d963673\" (UID: \"1c508601-c5f7-440f-b605-fb035d963673\") " Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793073 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793095 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793109 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793122 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793135 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793148 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793160 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793175 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c508601-c5f7-440f-b605-fb035d963673-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793188 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.793466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.793421 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-data" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.794432 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.794398 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-kube-api-access-9l27f" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "kube-api-access-9l27f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:51.794491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.794423 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-envoy" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.794491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.794460 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-kube-api-access-tjj8p" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "kube-api-access-tjj8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:51.794657 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.794540 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-envoy" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:45:51.795862 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.795826 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-token" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:51.795967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.795906 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-podinfo" pod "0c9632c7-52f5-4223-aeb4-4b2fcef13522" (UID: "0c9632c7-52f5-4223-aeb4-4b2fcef13522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:51.796019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.795991 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-istio-token" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:45:51.796019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.796007 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c508601-c5f7-440f-b605-fb035d963673-istio-podinfo" pod "1c508601-c5f7-440f-b605-fb035d963673" (UID: "1c508601-c5f7-440f-b605-fb035d963673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:45:51.894804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894767 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c508601-c5f7-440f-b605-fb035d963673-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.894804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894799 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.894804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894814 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.895047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894825 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.895047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894834 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.895047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894843 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.895047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894851 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l27f\" (UniqueName: \"kubernetes.io/projected/0c9632c7-52f5-4223-aeb4-4b2fcef13522-kube-api-access-9l27f\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.895047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894861 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c508601-c5f7-440f-b605-fb035d963673-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:51.895047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:51.894870 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjj8p\" (UniqueName: \"kubernetes.io/projected/1c508601-c5f7-440f-b605-fb035d963673-kube-api-access-tjj8p\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:45:52.581182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.577307 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" event={"ID":"0c9632c7-52f5-4223-aeb4-4b2fcef13522","Type":"ContainerDied","Data":"b1010149957169add564b40f438fb2aec20bc2f29fc16808adc0d97d955bad32"} Nov 02 21:45:52.581182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.577425 4510 scope.go:122] "RemoveContainer" containerID="f34cc388568f418d2e72e068439e1f6a9bbec26244ec96a35b22b2edd8ff65e9" Nov 02 21:45:52.581182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.577661 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k" Nov 02 21:45:52.583928 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.583895 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" Nov 02 21:45:52.584025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.583931 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh" event={"ID":"1c508601-c5f7-440f-b605-fb035d963673","Type":"ContainerDied","Data":"ab8fe69ed138a5de641bb2f007c69de1a26234ed1218c8fa7fd03d8d3769683f"} Nov 02 21:45:52.594085 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.594056 4510 scope.go:122] "RemoveContainer" containerID="8f35793f12da742476be6263f2e0269b9ef00f9d6b79fafa4f7e53b8e4fadf50" Nov 02 21:45:52.653992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.653958 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:45:52.654151 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.654023 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 21:45:52.693109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.693054 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k"] Nov 02 21:45:52.752292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.752247 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-n2h8k"] Nov 02 21:45:52.808779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.808742 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh"] Nov 02 21:45:52.814755 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:52.814725 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lq8nh"] Nov 02 21:45:53.654106 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:53.654068 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:45:53.654509 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:53.654140 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 21:45:54.162751 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:54.162721 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" path="/var/lib/kubelet/pods/0c9632c7-52f5-4223-aeb4-4b2fcef13522/volumes" Nov 02 21:45:54.163271 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:54.163256 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c508601-c5f7-440f-b605-fb035d963673" path="/var/lib/kubelet/pods/1c508601-c5f7-440f-b605-fb035d963673/volumes" Nov 02 21:45:54.653695 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:54.653660 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:45:54.653887 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:54.653715 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 21:45:55.654083 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:55.654043 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:45:55.654454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:55.654102 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 21:45:56.653926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:56.653839 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 02 21:45:56.653926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:56.653912 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 02 21:45:57.659174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:57.659143 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:57.659687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:57.659553 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:45:57.660350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:45:57.660332 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:46:01.344740 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.344709 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k"] Nov 02 21:46:01.345674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.345654 4510 cpu_manager.go:414] "containerMap: removing container" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" Nov 02 21:46:01.345757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.345675 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" Nov 02 21:46:01.345757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.345690 4510 cpu_manager.go:414] "containerMap: removing container" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" Nov 02 21:46:01.345757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.345695 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" Nov 02 21:46:01.346030 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.346002 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9632c7-52f5-4223-aeb4-4b2fcef13522" containerName="istio-proxy" Nov 02 21:46:01.346102 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.346039 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c508601-c5f7-440f-b605-fb035d963673" containerName="istio-proxy" Nov 02 21:46:01.352476 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.352453 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.390922 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.390888 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc"] Nov 02 21:46:01.395657 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.395635 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.408434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.408407 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k"] Nov 02 21:46:01.460945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.460905 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc"] Nov 02 21:46:01.560903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.560871 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561086 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.560908 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561086 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.560928 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561117 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561221 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561155 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561261 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bd5g\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-kube-api-access-8bd5g\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561291 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561418 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561395 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561461 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561433 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561460 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561597 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561653 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561630 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561700 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561782 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561753 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.561824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561794 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561856 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vd2\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-kube-api-access-s4vd2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561886 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.561914 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.561906 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663063 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663037 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663072 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663090 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663147 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vd2\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-kube-api-access-s4vd2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663174 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663191 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663234 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663260 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663284 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663370 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663395 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663459 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bd5g\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-kube-api-access-8bd5g\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663484 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663587 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663626 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663654 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663657 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663679 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.663930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663756 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.663930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663795 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.664301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663935 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.664301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663983 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.664301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.663935 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.664301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.664196 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.664301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.664200 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.664301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.664248 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.664494 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.664435 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.664698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.664678 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.666080 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.666059 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.666358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.666337 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.666408 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.666337 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.666408 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.666385 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.715968 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.715938 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vd2\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-kube-api-access-s4vd2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.716161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.716142 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.716405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.716383 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-pjj8k\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:01.716571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.716553 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bd5g\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-kube-api-access-8bd5g\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-mwdzc\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:01.968095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:01.968018 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:02.012189 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.011949 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:02.400105 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.400069 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k"] Nov 02 21:46:02.405064 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:46:02.405031 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1009d2_97e7_46dc_8e4c_0c297b269aab.slice/crio-8a7f62cc26986fe4ebe82c18731dc66eae2f599314af393442c5f232f5170a26 WatchSource:0}: Error finding container 8a7f62cc26986fe4ebe82c18731dc66eae2f599314af393442c5f232f5170a26: Status 404 returned error can't find the container with id 8a7f62cc26986fe4ebe82c18731dc66eae2f599314af393442c5f232f5170a26 Nov 02 21:46:02.407974 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.407941 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:02.408055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.408008 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:02.408055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.408036 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:02.445133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.445110 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc"] Nov 02 21:46:02.447298 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:46:02.447273 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b54d4fe_b9a2_4a9a_9c5e_438a75c10e6c.slice/crio-f8dc40ad8fe8dbbd2197c9467a4390caddae44181e7348164b1ac11ed3d21947 WatchSource:0}: Error finding container f8dc40ad8fe8dbbd2197c9467a4390caddae44181e7348164b1ac11ed3d21947: Status 404 returned error can't find the container with id f8dc40ad8fe8dbbd2197c9467a4390caddae44181e7348164b1ac11ed3d21947 Nov 02 21:46:02.450023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.449994 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:02.450103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.450056 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:02.450103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.450087 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:02.662829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.662731 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" event={"ID":"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c","Type":"ContainerStarted","Data":"c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2"} Nov 02 21:46:02.662829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.662801 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" event={"ID":"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c","Type":"ContainerStarted","Data":"f8dc40ad8fe8dbbd2197c9467a4390caddae44181e7348164b1ac11ed3d21947"} Nov 02 21:46:02.665013 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.664985 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" event={"ID":"3e1009d2-97e7-46dc-8e4c-0c297b269aab","Type":"ContainerStarted","Data":"0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac"} Nov 02 21:46:02.665148 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.665056 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" event={"ID":"3e1009d2-97e7-46dc-8e4c-0c297b269aab","Type":"ContainerStarted","Data":"8a7f62cc26986fe4ebe82c18731dc66eae2f599314af393442c5f232f5170a26"} Nov 02 21:46:02.772913 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.772857 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podStartSLOduration=1.772842683 podStartE2EDuration="1.772842683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:02.771233085 +0000 UTC m=+1988.747530884" watchObservedRunningTime="2026-11-02 21:46:02.772842683 +0000 UTC m=+1988.749140458" Nov 02 21:46:02.883074 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.883020 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podStartSLOduration=1.883005067 podStartE2EDuration="1.883005067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:02.880331969 +0000 UTC m=+1988.856629745" watchObservedRunningTime="2026-11-02 21:46:02.883005067 +0000 UTC m=+1988.859302842" Nov 02 21:46:02.968742 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:02.968665 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:03.013635 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:03.013590 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:03.015356 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:03.015323 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:03.015500 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:03.015399 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:03.971954 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:03.971912 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:46:03.972573 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:03.972540 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 21:46:04.013135 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:04.013102 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:04.013272 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:04.013150 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:04.971338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:04.970894 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:46:04.971338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:04.970954 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 21:46:05.013126 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:05.013080 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:05.013512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:05.013147 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:05.973550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:05.972968 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:46:05.973550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:05.973046 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 21:46:06.012827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:06.012789 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:06.012979 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:06.012850 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:06.973546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:06.968875 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:46:06.973546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:06.968934 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 21:46:07.012745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:07.012702 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:07.012904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:07.012770 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:07.969582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:07.969075 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:46:07.969582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:07.969146 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 21:46:08.012943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:08.012912 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:08.013377 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:08.013354 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:08.968455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:08.968418 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:46:08.968656 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:08.968489 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 21:46:09.012992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:09.012967 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:09.013345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:09.013009 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:09.968881 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:09.968845 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 21:46:09.969088 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:09.968899 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 21:46:10.013441 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:10.013402 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 21:46:10.013974 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:10.013449 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 21:46:11.191598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:11.188969 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:11.195591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:11.193187 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:11.201387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:11.201346 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:46:11.242723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:11.242694 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:11.243735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:11.243708 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:11.244568 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:11.244546 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:46:15.540215 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:46:15.540176 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:46:19.579831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.579792 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb"] Nov 02 21:46:19.585651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.585629 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.685984 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686208 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686064 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686208 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686137 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686208 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686193 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686324 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686236 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686364 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686330 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0192173-9c63-4856-802e-48190aae0942-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686411 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686391 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0192173-9c63-4856-802e-48190aae0942-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686440 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kch2q\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-kube-api-access-kch2q\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.686580 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.686564 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.770948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.770916 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb"] Nov 02 21:46:19.788351 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788323 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.788569 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788541 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.788659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788574 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.788659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788625 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.788767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788656 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.788858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788838 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.788988 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788969 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.789064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.788764 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.789064 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.789036 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.789195 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.789125 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0192173-9c63-4856-802e-48190aae0942-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.789195 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.789164 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0192173-9c63-4856-802e-48190aae0942-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.789302 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.789213 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kch2q\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-kube-api-access-kch2q\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.789385 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.789362 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.790286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.790262 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0192173-9c63-4856-802e-48190aae0942-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.792457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.791698 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.793109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.793083 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0192173-9c63-4856-802e-48190aae0942-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.841893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.841824 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kch2q\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-kube-api-access-kch2q\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.841893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.841823 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:19.904040 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:19.904010 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:20.358186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:20.358150 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb"] Nov 02 21:46:20.364515 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:46:20.364491 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0192173_9c63_4856_802e_48190aae0942.slice/crio-a995d7e99697de2d5df2608781b9d14b976f8206192079c2237e5d085c4682c5 WatchSource:0}: Error finding container a995d7e99697de2d5df2608781b9d14b976f8206192079c2237e5d085c4682c5: Status 404 returned error can't find the container with id a995d7e99697de2d5df2608781b9d14b976f8206192079c2237e5d085c4682c5 Nov 02 21:46:20.367301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:20.367263 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.367410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:20.367348 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.367410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:20.367392 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:20.820589 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:20.820551 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" event={"ID":"a0192173-9c63-4856-802e-48190aae0942","Type":"ContainerStarted","Data":"94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15"} Nov 02 21:46:20.820983 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:20.820620 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" event={"ID":"a0192173-9c63-4856-802e-48190aae0942","Type":"ContainerStarted","Data":"a995d7e99697de2d5df2608781b9d14b976f8206192079c2237e5d085c4682c5"} Nov 02 21:46:20.904513 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:20.904460 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:21.906319 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:21.906290 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:46:21.906723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:21.906337 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 21:46:22.588119 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:22.588059 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" podStartSLOduration=3.588044296 podStartE2EDuration="3.588044296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:20.940853773 +0000 UTC m=+2006.917151549" watchObservedRunningTime="2026-11-02 21:46:22.588044296 +0000 UTC m=+2008.564342070" Nov 02 21:46:22.591514 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:22.591486 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq"] Nov 02 21:46:22.591762 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:22.591734 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" containerID="cri-o://e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6" gracePeriod=30 Nov 02 21:46:22.904751 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:22.904720 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:46:22.904936 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:22.904783 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 21:46:23.905468 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:23.905440 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:46:23.906142 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:23.906110 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 21:46:24.865829 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:24.865788 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:46:24.866038 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:24.865850 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:46:24.904686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:24.904650 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:46:24.904809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:24.904707 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 21:46:25.904809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:25.904775 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 21:46:25.905197 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:25.904829 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 21:46:26.910123 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:26.910092 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:26.910583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:26.910565 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:26.911197 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:26.911172 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:46:27.862959 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.862940 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:46:27.896818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.896787 4510 generic.go:356] "Generic (PLEG): container finished" podID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerID="e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6" exitCode=0 Nov 02 21:46:27.896977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.896900 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" Nov 02 21:46:27.896977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.896931 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" event={"ID":"d4947166-47c6-4322-a3d5-c9e4ec483dc6","Type":"ContainerDied","Data":"e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6"} Nov 02 21:46:27.897100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.897001 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq" event={"ID":"d4947166-47c6-4322-a3d5-c9e4ec483dc6","Type":"ContainerDied","Data":"026adc65d67cf700a090e6a6ad8bbaba047561af0a20f7f835c4c41d886849f9"} Nov 02 21:46:27.897100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.897035 4510 scope.go:122] "RemoveContainer" containerID="e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6" Nov 02 21:46:27.907431 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.907413 4510 scope.go:122] "RemoveContainer" containerID="e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6" Nov 02 21:46:27.907731 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:46:27.907707 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6\": container with ID starting with e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6 not found: ID does not exist" containerID="e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6" Nov 02 21:46:27.907812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.907741 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6"} err="failed to get container status \"e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6\": rpc error: code = NotFound desc = could not find container \"e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6\": container with ID starting with e192620b80bcdf412001bbd3b886867bb5afbfbfe4140bd5f26f164332be92a6 not found: ID does not exist" Nov 02 21:46:27.938863 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.938837 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-certs\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.938887 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-kube-api-access-cxchb\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-kube-api-access-cxchb\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.938938 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istiod-ca-cert\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.938958 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-socket\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939026 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-data\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939137 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-podinfo\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939156 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-certs" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939197 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-envoy\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.939264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939249 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-token\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-token\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.940078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939300 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-socket" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:27.940078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939347 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-credential-socket\") pod \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\" (UID: \"d4947166-47c6-4322-a3d5-c9e4ec483dc6\") " Nov 02 21:46:27.940078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939351 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-data" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:27.940078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939451 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istiod-ca-cert" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:46:27.940078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.939675 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-credential-socket" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:27.940366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.940220 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:27.940366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.940239 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:27.940366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.940249 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:27.940366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.940260 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:27.940366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.940273 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:27.941391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.941367 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-podinfo" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:46:27.941499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.941425 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-envoy" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:46:27.941671 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.941646 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-token" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:27.941775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:27.941689 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-kube-api-access-cxchb" pod "d4947166-47c6-4322-a3d5-c9e4ec483dc6" (UID: "d4947166-47c6-4322-a3d5-c9e4ec483dc6"). InnerVolumeSpecName "kube-api-access-cxchb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:46:28.042266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:28.042228 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxchb\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-kube-api-access-cxchb\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:28.042266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:28.042256 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:28.042266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:28.042266 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:28.042266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:28.042274 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4947166-47c6-4322-a3d5-c9e4ec483dc6-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:46:28.304632 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:28.304594 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq"] Nov 02 21:46:28.309740 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:28.309711 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-rhzmq"] Nov 02 21:46:30.157047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:30.157010 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" path="/var/lib/kubelet/pods/d4947166-47c6-4322-a3d5-c9e4ec483dc6/volumes" Nov 02 21:46:34.674910 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.674872 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr"] Nov 02 21:46:34.675883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.675859 4510 cpu_manager.go:414] "containerMap: removing container" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" Nov 02 21:46:34.675883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.675883 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" Nov 02 21:46:34.676159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.676143 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4947166-47c6-4322-a3d5-c9e4ec483dc6" containerName="istio-proxy" Nov 02 21:46:34.681386 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.681365 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.777822 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.777785 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr"] Nov 02 21:46:34.837903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.837873 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.837917 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.838010 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c2a84c-9317-4577-953a-2191b2874cc9-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.838054 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.838135 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c2a84c-9317-4577-953a-2191b2874cc9-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.838210 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vv6\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-kube-api-access-d6vv6\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838261 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.838253 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.838290 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.838453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.838441 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.940740 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.940660 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.940889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.940737 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.940889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.940789 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.940889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.940846 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c2a84c-9317-4577-953a-2191b2874cc9-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.940919 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.940983 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c2a84c-9317-4577-953a-2191b2874cc9-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.941096 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vv6\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-kube-api-access-d6vv6\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.941114 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-credential-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.941138 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.941194 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941630 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.941607 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-data\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941840 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.941818 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c2a84c-9317-4577-953a-2191b2874cc9-istiod-ca-cert\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.941840 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.941827 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-socket\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.942690 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.942667 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-certs\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.943413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.943388 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-envoy\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.943560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.943538 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c2a84c-9317-4577-953a-2191b2874cc9-istio-podinfo\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.994098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.994063 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-istio-token\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.994257 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.994191 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vv6\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-kube-api-access-d6vv6\") pod \"e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:34.994857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:34.994837 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:35.435041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:35.435005 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr"] Nov 02 21:46:35.439999 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:46:35.439970 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c2a84c_9317_4577_953a_2191b2874cc9.slice/crio-891e6fb0f5a1afc897ab70b9421410e15736b47849c8290962db188e9d51f0df WatchSource:0}: Error finding container 891e6fb0f5a1afc897ab70b9421410e15736b47849c8290962db188e9d51f0df: Status 404 returned error can't find the container with id 891e6fb0f5a1afc897ab70b9421410e15736b47849c8290962db188e9d51f0df Nov 02 21:46:35.442821 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:35.442788 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:35.442882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:35.442858 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:35.442922 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:35.442886 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:46:35.953473 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:35.953439 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" event={"ID":"83c2a84c-9317-4577-953a-2191b2874cc9","Type":"ContainerStarted","Data":"7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd"} Nov 02 21:46:35.953977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:35.953493 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" event={"ID":"83c2a84c-9317-4577-953a-2191b2874cc9","Type":"ContainerStarted","Data":"891e6fb0f5a1afc897ab70b9421410e15736b47849c8290962db188e9d51f0df"} Nov 02 21:46:35.995792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:35.995768 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:36.063851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:36.063804 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" podStartSLOduration=2.06378929 podStartE2EDuration="2.06378929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:46:36.061189777 +0000 UTC m=+2022.037487554" watchObservedRunningTime="2026-11-02 21:46:36.06378929 +0000 UTC m=+2022.040087064" Nov 02 21:46:36.997063 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:36.997030 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:46:36.997457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:36.997093 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 21:46:37.052143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:37.052105 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:46:38.004539 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:38.001448 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:46:38.004539 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:38.001512 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 21:46:38.996000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:38.995968 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:46:38.996185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:38.996019 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 21:46:39.996118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:39.996084 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:46:39.996724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:39.996697 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 21:46:40.995243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:40.995209 4510 patch_prober.go:29] interesting pod/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 21:46:40.995418 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:40.995269 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 21:46:41.999772 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:41.999739 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:42.000251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:42.000230 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:42.000998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:42.000977 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:46:45.534707 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:46:45.534666 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:46:59.698149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:59.698117 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p"] Nov 02 21:46:59.698723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:46:59.698324 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" podUID="4e4665b8-c27f-44ef-8114-4e692d3dd622" containerName="maas-api" containerID="cri-o://344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85" gracePeriod=30 Nov 02 21:47:00.001719 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.001694 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:47:00.080199 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.080173 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4665b8-c27f-44ef-8114-4e692d3dd622-kube-api-access-9lh7z\" (UniqueName: \"kubernetes.io/projected/4e4665b8-c27f-44ef-8114-4e692d3dd622-kube-api-access-9lh7z\") pod \"4e4665b8-c27f-44ef-8114-4e692d3dd622\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " Nov 02 21:47:00.080353 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.080227 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-metrics-certs\") pod \"4e4665b8-c27f-44ef-8114-4e692d3dd622\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " Nov 02 21:47:00.080353 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.080296 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4665b8-c27f-44ef-8114-4e692d3dd622-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4665b8-c27f-44ef-8114-4e692d3dd622-openshift-service-ca\") pod \"4e4665b8-c27f-44ef-8114-4e692d3dd622\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " Nov 02 21:47:00.080462 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.080446 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls\") pod \"4e4665b8-c27f-44ef-8114-4e692d3dd622\" (UID: \"4e4665b8-c27f-44ef-8114-4e692d3dd622\") " Nov 02 21:47:00.080764 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.080733 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4665b8-c27f-44ef-8114-4e692d3dd622-openshift-service-ca" pod "4e4665b8-c27f-44ef-8114-4e692d3dd622" (UID: "4e4665b8-c27f-44ef-8114-4e692d3dd622"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:47:00.081249 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.081226 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e4665b8-c27f-44ef-8114-4e692d3dd622-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:00.082581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.082557 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4665b8-c27f-44ef-8114-4e692d3dd622-kube-api-access-9lh7z" pod "4e4665b8-c27f-44ef-8114-4e692d3dd622" (UID: "4e4665b8-c27f-44ef-8114-4e692d3dd622"). InnerVolumeSpecName "kube-api-access-9lh7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:00.082772 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.082755 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls" pod "4e4665b8-c27f-44ef-8114-4e692d3dd622" (UID: "4e4665b8-c27f-44ef-8114-4e692d3dd622"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:00.082846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.082798 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-metrics-certs" pod "4e4665b8-c27f-44ef-8114-4e692d3dd622" (UID: "4e4665b8-c27f-44ef-8114-4e692d3dd622"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:00.198099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.198051 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:00.198099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.198086 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lh7z\" (UniqueName: \"kubernetes.io/projected/4e4665b8-c27f-44ef-8114-4e692d3dd622-kube-api-access-9lh7z\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:00.198099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.198101 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e4665b8-c27f-44ef-8114-4e692d3dd622-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:00.203873 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.203839 4510 generic.go:356] "Generic (PLEG): container finished" podID="4e4665b8-c27f-44ef-8114-4e692d3dd622" containerID="344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85" exitCode=0 Nov 02 21:47:00.204051 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.204000 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" event={"ID":"4e4665b8-c27f-44ef-8114-4e692d3dd622","Type":"ContainerDied","Data":"344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85"} Nov 02 21:47:00.204051 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.204044 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" event={"ID":"4e4665b8-c27f-44ef-8114-4e692d3dd622","Type":"ContainerDied","Data":"6324f74fdabcb2d576d61d592838de4e20e9c18aa4731d48b61c458e1c922558"} Nov 02 21:47:00.204141 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.204072 4510 scope.go:122] "RemoveContainer" containerID="344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85" Nov 02 21:47:00.204220 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.204207 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p" Nov 02 21:47:00.220177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.220154 4510 scope.go:122] "RemoveContainer" containerID="344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85" Nov 02 21:47:00.220570 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:47:00.220540 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85\": container with ID starting with 344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85 not found: ID does not exist" containerID="344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85" Nov 02 21:47:00.220659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.220582 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85"} err="failed to get container status \"344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85\": rpc error: code = NotFound desc = could not find container \"344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85\": container with ID starting with 344daab920bd55ca7570e04608d81ed42685ef15c2103b8d7013bf030f11ca85 not found: ID does not exist" Nov 02 21:47:00.358867 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.358835 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p"] Nov 02 21:47:00.363982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:00.363948 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-06366c-9cbf87875-gwc9p"] Nov 02 21:47:02.157226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:02.157190 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4665b8-c27f-44ef-8114-4e692d3dd622" path="/var/lib/kubelet/pods/4e4665b8-c27f-44ef-8114-4e692d3dd622/volumes" Nov 02 21:47:15.532211 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:47:15.532178 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:47:21.352247 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.352213 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc"] Nov 02 21:47:21.352750 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.352433 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" podUID="7039890a-43f4-4d63-bc88-1a9d2c54f413" containerName="maas-api" containerID="cri-o://f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5" gracePeriod=30 Nov 02 21:47:21.656224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.656190 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:47:21.821402 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.821365 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-metrics-certs\") pod \"7039890a-43f4-4d63-bc88-1a9d2c54f413\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " Nov 02 21:47:21.821972 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.821604 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7039890a-43f4-4d63-bc88-1a9d2c54f413-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7039890a-43f4-4d63-bc88-1a9d2c54f413-openshift-service-ca\") pod \"7039890a-43f4-4d63-bc88-1a9d2c54f413\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " Nov 02 21:47:21.821972 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.821699 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7039890a-43f4-4d63-bc88-1a9d2c54f413-kube-api-access-9vpzf\" (UniqueName: \"kubernetes.io/projected/7039890a-43f4-4d63-bc88-1a9d2c54f413-kube-api-access-9vpzf\") pod \"7039890a-43f4-4d63-bc88-1a9d2c54f413\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " Nov 02 21:47:21.821972 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.821852 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls\") pod \"7039890a-43f4-4d63-bc88-1a9d2c54f413\" (UID: \"7039890a-43f4-4d63-bc88-1a9d2c54f413\") " Nov 02 21:47:21.822282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.822253 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7039890a-43f4-4d63-bc88-1a9d2c54f413-openshift-service-ca" pod "7039890a-43f4-4d63-bc88-1a9d2c54f413" (UID: "7039890a-43f4-4d63-bc88-1a9d2c54f413"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:47:21.822667 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.822641 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7039890a-43f4-4d63-bc88-1a9d2c54f413-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:21.824943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.824913 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls" pod "7039890a-43f4-4d63-bc88-1a9d2c54f413" (UID: "7039890a-43f4-4d63-bc88-1a9d2c54f413"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:21.828937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.828898 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-metrics-certs" pod "7039890a-43f4-4d63-bc88-1a9d2c54f413" (UID: "7039890a-43f4-4d63-bc88-1a9d2c54f413"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:47:21.833004 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.832975 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7039890a-43f4-4d63-bc88-1a9d2c54f413-kube-api-access-9vpzf" pod "7039890a-43f4-4d63-bc88-1a9d2c54f413" (UID: "7039890a-43f4-4d63-bc88-1a9d2c54f413"). InnerVolumeSpecName "kube-api-access-9vpzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:47:21.935598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.926539 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpzf\" (UniqueName: \"kubernetes.io/projected/7039890a-43f4-4d63-bc88-1a9d2c54f413-kube-api-access-9vpzf\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:21.935598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.926578 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:21.935598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:21.926592 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7039890a-43f4-4d63-bc88-1a9d2c54f413-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:47:22.444560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.442656 4510 generic.go:356] "Generic (PLEG): container finished" podID="7039890a-43f4-4d63-bc88-1a9d2c54f413" containerID="f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5" exitCode=0 Nov 02 21:47:22.444560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.442722 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" Nov 02 21:47:22.444560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.442726 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" event={"ID":"7039890a-43f4-4d63-bc88-1a9d2c54f413","Type":"ContainerDied","Data":"f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5"} Nov 02 21:47:22.444560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.442851 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc" event={"ID":"7039890a-43f4-4d63-bc88-1a9d2c54f413","Type":"ContainerDied","Data":"4a76196b0d5e26edec32083409d3aebc9b0f46965397bd682d49efbbfacf0772"} Nov 02 21:47:22.444560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.442875 4510 scope.go:122] "RemoveContainer" containerID="f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5" Nov 02 21:47:22.457174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.457148 4510 scope.go:122] "RemoveContainer" containerID="f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5" Nov 02 21:47:22.457778 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:47:22.457744 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5\": container with ID starting with f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5 not found: ID does not exist" containerID="f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5" Nov 02 21:47:22.457963 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.457934 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5"} err="failed to get container status \"f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5\": rpc error: code = NotFound desc = could not find container \"f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5\": container with ID starting with f89707d2ebf502c93e41157a09d738568aab9bae5a4e619e67e4c18921f04cd5 not found: ID does not exist" Nov 02 21:47:22.597807 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.597773 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc"] Nov 02 21:47:22.644058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:22.644020 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-j88vc"] Nov 02 21:47:24.157494 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:24.157461 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7039890a-43f4-4d63-bc88-1a9d2c54f413" path="/var/lib/kubelet/pods/7039890a-43f4-4d63-bc88-1a9d2c54f413/volumes" Nov 02 21:47:45.535920 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:47:45.535879 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:47:48.829468 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:47:48.829436 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:48:00.312675 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.312633 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w"] Nov 02 21:48:00.313181 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.312849 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" podUID="dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" containerName="maas-api" containerID="cri-o://d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da" gracePeriod=30 Nov 02 21:48:00.635602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.635576 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:48:00.718010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.717974 4510 generic.go:356] "Generic (PLEG): container finished" podID="dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" containerID="d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da" exitCode=0 Nov 02 21:48:00.718180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.718027 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" Nov 02 21:48:00.718180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.718053 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" event={"ID":"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2","Type":"ContainerDied","Data":"d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da"} Nov 02 21:48:00.718180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.718113 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w" event={"ID":"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2","Type":"ContainerDied","Data":"189b80e0037955fc49aa89ae0a2ee85d5b4bbf18e7aa5af32ffd143cfae606eb"} Nov 02 21:48:00.718180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.718142 4510 scope.go:122] "RemoveContainer" containerID="d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da" Nov 02 21:48:00.721647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.721626 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-openshift-service-ca\") pod \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " Nov 02 21:48:00.721771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.721726 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls\") pod \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " Nov 02 21:48:00.721833 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.721775 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-metrics-certs\") pod \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " Nov 02 21:48:00.721953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.721940 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-kube-api-access-w2fd9\" (UniqueName: \"kubernetes.io/projected/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-kube-api-access-w2fd9\") pod \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\" (UID: \"dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2\") " Nov 02 21:48:00.722010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.721977 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-openshift-service-ca" pod "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" (UID: "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:00.722801 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.722777 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:00.723931 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.723906 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls" pod "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" (UID: "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:00.724291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.724267 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-kube-api-access-w2fd9" pod "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" (UID: "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2"). InnerVolumeSpecName "kube-api-access-w2fd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:00.724911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.724894 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-metrics-certs" pod "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" (UID: "dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:00.735051 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.735030 4510 scope.go:122] "RemoveContainer" containerID="d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da" Nov 02 21:48:00.735350 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:00.735327 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da\": container with ID starting with d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da not found: ID does not exist" containerID="d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da" Nov 02 21:48:00.735421 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.735359 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da"} err="failed to get container status \"d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da\": rpc error: code = NotFound desc = could not find container \"d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da\": container with ID starting with d1a446394543e2206abd9652060349bfc700390ca20dc1219aeb635f033572da not found: ID does not exist" Nov 02 21:48:00.824945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.824902 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2fd9\" (UniqueName: \"kubernetes.io/projected/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-kube-api-access-w2fd9\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:00.824945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.824943 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:00.824945 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:00.824959 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:01.132961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:01.132929 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w"] Nov 02 21:48:01.179042 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:01.179008 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-9f77b7-558b6b84dd-w6r5w"] Nov 02 21:48:02.157038 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:02.157000 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" path="/var/lib/kubelet/pods/dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2/volumes" Nov 02 21:48:06.210834 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.210764 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp"] Nov 02 21:48:06.211282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.210947 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" podUID="c74e302b-f55c-4158-a0d3-8a682e6c75eb" containerName="maas-api" containerID="cri-o://54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845" gracePeriod=30 Nov 02 21:48:06.521744 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.521715 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:48:06.610158 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.610120 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74e302b-f55c-4158-a0d3-8a682e6c75eb-kube-api-access-jmr7c\" (UniqueName: \"kubernetes.io/projected/c74e302b-f55c-4158-a0d3-8a682e6c75eb-kube-api-access-jmr7c\") pod \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.611713 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74e302b-f55c-4158-a0d3-8a682e6c75eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74e302b-f55c-4158-a0d3-8a682e6c75eb-openshift-service-ca\") pod \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.611880 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls\") pod \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.611925 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-metrics-certs\") pod \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\" (UID: \"c74e302b-f55c-4158-a0d3-8a682e6c75eb\") " Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.612122 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74e302b-f55c-4158-a0d3-8a682e6c75eb-openshift-service-ca" pod "c74e302b-f55c-4158-a0d3-8a682e6c75eb" (UID: "c74e302b-f55c-4158-a0d3-8a682e6c75eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.612724 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74e302b-f55c-4158-a0d3-8a682e6c75eb-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.614408 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74e302b-f55c-4158-a0d3-8a682e6c75eb-kube-api-access-jmr7c" pod "c74e302b-f55c-4158-a0d3-8a682e6c75eb" (UID: "c74e302b-f55c-4158-a0d3-8a682e6c75eb"). InnerVolumeSpecName "kube-api-access-jmr7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.615391 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls" pod "c74e302b-f55c-4158-a0d3-8a682e6c75eb" (UID: "c74e302b-f55c-4158-a0d3-8a682e6c75eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:06.615689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.615606 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-metrics-certs" pod "c74e302b-f55c-4158-a0d3-8a682e6c75eb" (UID: "c74e302b-f55c-4158-a0d3-8a682e6c75eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:06.714133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.714097 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmr7c\" (UniqueName: \"kubernetes.io/projected/c74e302b-f55c-4158-a0d3-8a682e6c75eb-kube-api-access-jmr7c\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:06.714133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.714130 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:06.714133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.714141 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74e302b-f55c-4158-a0d3-8a682e6c75eb-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:06.769905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.769812 4510 generic.go:356] "Generic (PLEG): container finished" podID="c74e302b-f55c-4158-a0d3-8a682e6c75eb" containerID="54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845" exitCode=0 Nov 02 21:48:06.770076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.769916 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" event={"ID":"c74e302b-f55c-4158-a0d3-8a682e6c75eb","Type":"ContainerDied","Data":"54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845"} Nov 02 21:48:06.770076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.770047 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" event={"ID":"c74e302b-f55c-4158-a0d3-8a682e6c75eb","Type":"ContainerDied","Data":"0eed6551e2ed8b35ca7f0ab5882a51de86f13987b3bc6ff36af7cfd8dab19d98"} Nov 02 21:48:06.770182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.770081 4510 scope.go:122] "RemoveContainer" containerID="54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845" Nov 02 21:48:06.770256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.769878 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp" Nov 02 21:48:06.786395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.786373 4510 scope.go:122] "RemoveContainer" containerID="54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845" Nov 02 21:48:06.786729 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:06.786703 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845\": container with ID starting with 54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845 not found: ID does not exist" containerID="54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845" Nov 02 21:48:06.786860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.786740 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845"} err="failed to get container status \"54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845\": rpc error: code = NotFound desc = could not find container \"54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845\": container with ID starting with 54e3fbd3bb489d0f786070e43f47abf2fe9cb08a49b786d09d4a0875bfd75845 not found: ID does not exist" Nov 02 21:48:06.925870 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.925842 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp"] Nov 02 21:48:06.930657 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:06.930623 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-d4d045-6478c75b6-qkznp"] Nov 02 21:48:08.156983 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:08.156946 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74e302b-f55c-4158-a0d3-8a682e6c75eb" path="/var/lib/kubelet/pods/c74e302b-f55c-4158-a0d3-8a682e6c75eb/volumes" Nov 02 21:48:08.762492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:08.762460 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd"] Nov 02 21:48:08.762706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:08.762682 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" containerID="cri-o://2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa" gracePeriod=30 Nov 02 21:48:13.706665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:13.706626 4510 patch_prober.go:29] interesting pod/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:48:13.707093 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:13.706681 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:48:14.060267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.060243 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:48:14.131293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131260 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-podinfo\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.131478 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131459 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istiod-ca-cert\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.131587 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131551 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-token\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-token\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.131688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131671 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-data\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.131745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131713 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-credential-socket\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.131804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131741 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-certs\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.131804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131772 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-kube-api-access-r5bld\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-kube-api-access-r5bld\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.131911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131841 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-socket\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.132024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131926 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istiod-ca-cert" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:14.132209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.131968 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-data" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:14.132209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.132196 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-socket" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:14.132209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.132047 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-envoy\") pod \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\" (UID: \"aa6d30ce-9a52-400e-a7c3-0937edd9f564\") " Nov 02 21:48:14.132209 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.132001 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-certs" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:14.132434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.132018 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-credential-socket" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:14.133034 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.133011 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.133112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.133042 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.133112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.133058 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.133112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.133076 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.133112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.133090 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.133848 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.133818 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-podinfo" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:48:14.134136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.134117 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-token" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:14.134136 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.134122 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-kube-api-access-r5bld" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "kube-api-access-r5bld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:14.134489 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.134474 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-envoy" pod "aa6d30ce-9a52-400e-a7c3-0937edd9f564" (UID: "aa6d30ce-9a52-400e-a7c3-0937edd9f564"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:14.234660 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.234597 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.234660 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.234630 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5bld\" (UniqueName: \"kubernetes.io/projected/aa6d30ce-9a52-400e-a7c3-0937edd9f564-kube-api-access-r5bld\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.234901 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.234716 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.234901 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.234728 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa6d30ce-9a52-400e-a7c3-0937edd9f564-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:14.877665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.850052 4510 generic.go:356] "Generic (PLEG): container finished" podID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerID="2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa" exitCode=0 Nov 02 21:48:14.877665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.850226 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" event={"ID":"aa6d30ce-9a52-400e-a7c3-0937edd9f564","Type":"ContainerDied","Data":"2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa"} Nov 02 21:48:14.877665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.850280 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" event={"ID":"aa6d30ce-9a52-400e-a7c3-0937edd9f564","Type":"ContainerDied","Data":"60907db633bfc5bd34906fbfa55d7fd45c2dbbfa90e93baee4151556bbdbb8ce"} Nov 02 21:48:14.877665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.850313 4510 scope.go:122] "RemoveContainer" containerID="2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa" Nov 02 21:48:14.877665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.850456 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd" Nov 02 21:48:14.936660 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.936621 4510 scope.go:122] "RemoveContainer" containerID="88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a" Nov 02 21:48:14.955812 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.955785 4510 scope.go:122] "RemoveContainer" containerID="2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa" Nov 02 21:48:14.956164 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:14.956132 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa\": container with ID starting with 2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa not found: ID does not exist" containerID="2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa" Nov 02 21:48:14.956250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.956178 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa"} err="failed to get container status \"2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa\": rpc error: code = NotFound desc = could not find container \"2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa\": container with ID starting with 2b6ee1bd372f851eaaafd36593bc64a9a0367b5cfdea4b4d711a6783e1a5c8aa not found: ID does not exist" Nov 02 21:48:14.956250 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.956205 4510 scope.go:122] "RemoveContainer" containerID="88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a" Nov 02 21:48:14.956507 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:14.956485 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a\": container with ID starting with 88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a not found: ID does not exist" containerID="88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a" Nov 02 21:48:14.956613 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:14.956514 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a"} err="failed to get container status \"88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a\": rpc error: code = NotFound desc = could not find container \"88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a\": container with ID starting with 88e479fcd07a20a1a9046469d8dcd628eb3e237315cddd999aa2f9fab74a957a not found: ID does not exist" Nov 02 21:48:15.025795 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:15.025768 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd"] Nov 02 21:48:15.029454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:15.029429 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-9f77b7-openshift-default-6c9975d87-xngcd"] Nov 02 21:48:15.532345 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:15.532308 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:48:16.155327 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:16.155296 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" path="/var/lib/kubelet/pods/aa6d30ce-9a52-400e-a7c3-0937edd9f564/volumes" Nov 02 21:48:23.252039 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:23.252002 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m"] Nov 02 21:48:23.252892 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:23.252218 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" containerID="cri-o://f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16" gracePeriod=30 Nov 02 21:48:23.795992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:23.795961 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb"] Nov 02 21:48:23.796200 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:23.796178 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" containerID="cri-o://cea251b13d9b9e00890eba1a649f5c352af2848d6f29cad5c6efa8740d9540f5" gracePeriod=30 Nov 02 21:48:24.657000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:24.656958 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp"] Nov 02 21:48:24.657475 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:24.657172 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" containerID="cri-o://f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad" gracePeriod=30 Nov 02 21:48:24.799265 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:24.799232 4510 patch_prober.go:29] interesting pod/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:48:24.799439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:24.799295 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:48:25.592381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.592345 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p"] Nov 02 21:48:25.592607 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.592587 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerName="discovery" containerID="cri-o://8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be" gracePeriod=30 Nov 02 21:48:25.629499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.629473 4510 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bxq4p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:8080/ready\": dial tcp 10.134.0.88:8080: connect: connection refused" start-of-body= Nov 02 21:48:25.629605 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.629570 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.88:8080/ready\": dial tcp 10.134.0.88:8080: connect: connection refused" Nov 02 21:48:25.882654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.882629 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:48:25.931805 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.931777 4510 generic.go:356] "Generic (PLEG): container finished" podID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerID="8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be" exitCode=0 Nov 02 21:48:25.931960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.931818 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" Nov 02 21:48:25.931960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.931855 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" event={"ID":"23ba9c8d-b0ea-437a-95fd-b4ce24677947","Type":"ContainerDied","Data":"8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be"} Nov 02 21:48:25.931960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.931904 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p" event={"ID":"23ba9c8d-b0ea-437a-95fd-b4ce24677947","Type":"ContainerDied","Data":"9f52515ac394f70efba91f34c25a61e84aff26058aca4b398f79d4a9262eb5a6"} Nov 02 21:48:25.931960 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.931924 4510 scope.go:122] "RemoveContainer" containerID="8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be" Nov 02 21:48:25.943280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.943257 4510 scope.go:122] "RemoveContainer" containerID="8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be" Nov 02 21:48:25.943602 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:25.943580 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be\": container with ID starting with 8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be not found: ID does not exist" containerID="8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be" Nov 02 21:48:25.943689 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:25.943614 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be"} err="failed to get container status \"8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be\": rpc error: code = NotFound desc = could not find container \"8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be\": container with ID starting with 8e88d9290bd7e577a854b00aec7b4025a8ec3eb6eb38fc3a1aa4f2a34565d2be not found: ID does not exist" Nov 02 21:48:26.020857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.020825 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-dns-cert\") pod \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " Nov 02 21:48:26.021016 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.020916 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-token\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-token\") pod \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " Nov 02 21:48:26.021131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.021117 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-cacerts\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-cacerts\") pod \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " Nov 02 21:48:26.021204 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.021141 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-ca-configmap\") pod \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " Nov 02 21:48:26.021204 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.021180 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-kube-api-access-ctbrg\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-kube-api-access-ctbrg\") pod \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " Nov 02 21:48:26.021321 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.021216 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-kubeconfig\") pod \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " Nov 02 21:48:26.021321 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.021280 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ba9c8d-b0ea-437a-95fd-b4ce24677947-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ba9c8d-b0ea-437a-95fd-b4ce24677947-local-certs\") pod \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\" (UID: \"23ba9c8d-b0ea-437a-95fd-b4ce24677947\") " Nov 02 21:48:26.021775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.021580 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-ca-configmap" pod "23ba9c8d-b0ea-437a-95fd-b4ce24677947" (UID: "23ba9c8d-b0ea-437a-95fd-b4ce24677947"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:26.021998 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.021973 4510 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-ca-configmap\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.023713 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.023687 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-dns-cert" pod "23ba9c8d-b0ea-437a-95fd-b4ce24677947" (UID: "23ba9c8d-b0ea-437a-95fd-b4ce24677947"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:26.023919 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.023876 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-cacerts" pod "23ba9c8d-b0ea-437a-95fd-b4ce24677947" (UID: "23ba9c8d-b0ea-437a-95fd-b4ce24677947"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:26.024011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.023967 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-kubeconfig" pod "23ba9c8d-b0ea-437a-95fd-b4ce24677947" (UID: "23ba9c8d-b0ea-437a-95fd-b4ce24677947"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:26.024179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.024157 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-token" pod "23ba9c8d-b0ea-437a-95fd-b4ce24677947" (UID: "23ba9c8d-b0ea-437a-95fd-b4ce24677947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:26.024224 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.024193 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-kube-api-access-ctbrg" pod "23ba9c8d-b0ea-437a-95fd-b4ce24677947" (UID: "23ba9c8d-b0ea-437a-95fd-b4ce24677947"). InnerVolumeSpecName "kube-api-access-ctbrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:26.024297 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.024281 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ba9c8d-b0ea-437a-95fd-b4ce24677947-local-certs" pod "23ba9c8d-b0ea-437a-95fd-b4ce24677947" (UID: "23ba9c8d-b0ea-437a-95fd-b4ce24677947"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:26.123673 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.123647 4510 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-csr-dns-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.123673 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.123672 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.123673 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.123681 4510 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-cacerts\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.123874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.123693 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctbrg\" (UniqueName: \"kubernetes.io/projected/23ba9c8d-b0ea-437a-95fd-b4ce24677947-kube-api-access-ctbrg\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.123874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.123703 4510 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/23ba9c8d-b0ea-437a-95fd-b4ce24677947-istio-kubeconfig\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.123874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.123711 4510 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ba9c8d-b0ea-437a-95fd-b4ce24677947-local-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:26.340692 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.340661 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p"] Nov 02 21:48:26.344409 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.344384 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bxq4p"] Nov 02 21:48:26.555319 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.555228 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:48:26.555319 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:26.555297 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:48:28.156140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.156104 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" path="/var/lib/kubelet/pods/23ba9c8d-b0ea-437a-95fd-b4ce24677947/volumes" Nov 02 21:48:28.537918 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.537885 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:48:28.551951 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.551916 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-token\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-token\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552088 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-credential-socket\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552124 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-certs\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552164 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552161 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-data\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552205 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istiod-ca-cert\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552241 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-kube-api-access-4bnzg\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-kube-api-access-4bnzg\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552280 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-socket\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552345 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-podinfo\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552388 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-credential-socket" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:28.552594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552426 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-certs" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:28.552594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552415 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-envoy\") pod \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\" (UID: \"c78ab5d8-f481-4ac1-9d2a-5553b735f385\") " Nov 02 21:48:28.552594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552476 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-data" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:28.552807 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552686 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-socket" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:28.553004 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.552908 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istiod-ca-cert" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:28.553330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.553253 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.553330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.553275 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.553330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.553288 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.553330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.553300 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.553330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.553312 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.555101 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.555072 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-envoy" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:28.555218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.555197 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-podinfo" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:48:28.555271 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.555235 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-kube-api-access-4bnzg" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "kube-api-access-4bnzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:28.555308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.555268 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-token" pod "c78ab5d8-f481-4ac1-9d2a-5553b735f385" (UID: "c78ab5d8-f481-4ac1-9d2a-5553b735f385"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:28.655168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.655136 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bnzg\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-kube-api-access-4bnzg\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.655168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.655167 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.655358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.655177 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.655358 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.655186 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c78ab5d8-f481-4ac1-9d2a-5553b735f385-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:28.789287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.789206 4510 patch_prober.go:29] interesting pod/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:48:28.789287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.789262 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:48:28.978437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.978402 4510 generic.go:356] "Generic (PLEG): container finished" podID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerID="cea251b13d9b9e00890eba1a649f5c352af2848d6f29cad5c6efa8740d9540f5" exitCode=0 Nov 02 21:48:28.978644 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.978473 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" event={"ID":"f5e897ad-5975-4ab4-a508-a574208a7aef","Type":"ContainerDied","Data":"cea251b13d9b9e00890eba1a649f5c352af2848d6f29cad5c6efa8740d9540f5"} Nov 02 21:48:28.980274 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.980247 4510 generic.go:356] "Generic (PLEG): container finished" podID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerID="f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16" exitCode=0 Nov 02 21:48:28.980416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.980400 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" event={"ID":"c78ab5d8-f481-4ac1-9d2a-5553b735f385","Type":"ContainerDied","Data":"f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16"} Nov 02 21:48:28.980487 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.980448 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" event={"ID":"c78ab5d8-f481-4ac1-9d2a-5553b735f385","Type":"ContainerDied","Data":"f8b019338cb0ad6475f1b55c1149aeeefdeeb9c496a6c77b7e25281a41c1ae33"} Nov 02 21:48:28.980591 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.980482 4510 scope.go:122] "RemoveContainer" containerID="f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16" Nov 02 21:48:28.980751 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:28.980704 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m" Nov 02 21:48:29.006312 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.006276 4510 scope.go:122] "RemoveContainer" containerID="f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16" Nov 02 21:48:29.006703 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:29.006672 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16\": container with ID starting with f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16 not found: ID does not exist" containerID="f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16" Nov 02 21:48:29.006878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.006715 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16"} err="failed to get container status \"f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16\": rpc error: code = NotFound desc = could not find container \"f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16\": container with ID starting with f7612997d593ce199a9d5c25ab7b8fc6f7d00e6b12514c0a0ee39d8746e6ce16 not found: ID does not exist" Nov 02 21:48:29.079286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.079262 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:48:29.136396 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.136362 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m"] Nov 02 21:48:29.140811 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.140778 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-5rd2m"] Nov 02 21:48:29.165185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165162 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-socket\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165211 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-podinfo\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165239 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-data\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165271 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-kube-api-access-sclks\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-kube-api-access-sclks\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165415 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-envoy\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165463 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-token\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-token\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165468 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-socket" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165546 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-credential-socket\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165553 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-data" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165579 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e897ad-5975-4ab4-a508-a574208a7aef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e897ad-5975-4ab4-a508-a574208a7aef-istiod-ca-cert\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.165650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165611 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-certs\") pod \"f5e897ad-5975-4ab4-a508-a574208a7aef\" (UID: \"f5e897ad-5975-4ab4-a508-a574208a7aef\") " Nov 02 21:48:29.166182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.165991 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e897ad-5975-4ab4-a508-a574208a7aef-istiod-ca-cert" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:29.166182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.166104 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-credential-socket" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.166182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.166114 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-certs" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.166664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.166371 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.166664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.166432 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.166664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.166448 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.166664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.166462 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5e897ad-5975-4ab4-a508-a574208a7aef-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.166664 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.166475 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.168258 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.168239 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-podinfo" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:48:29.168344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.168308 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-envoy" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.168344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.168336 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-kube-api-access-sclks" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "kube-api-access-sclks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:29.168617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.168598 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-token" pod "f5e897ad-5975-4ab4-a508-a574208a7aef" (UID: "f5e897ad-5975-4ab4-a508-a574208a7aef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:29.268657 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.268621 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.268657 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.268656 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.268905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.268671 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5e897ad-5975-4ab4-a508-a574208a7aef-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.268905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.268686 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sclks\" (UniqueName: \"kubernetes.io/projected/f5e897ad-5975-4ab4-a508-a574208a7aef-kube-api-access-sclks\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.950921 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.950892 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:48:29.979373 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979345 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-kube-api-access-fqg6w\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-kube-api-access-fqg6w\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979497 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979432 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-credential-socket\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979497 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979473 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-envoy\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979501 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istiod-ca-cert\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979576 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-data\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979602 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-token\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-token\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979636 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-socket\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979709 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-podinfo\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979777 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-credential-socket" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.979861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979805 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-certs\") pod \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\" (UID: \"875f34a7-f57b-4c3b-a15b-a6a2df0f816e\") " Nov 02 21:48:29.979963 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.979945 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istiod-ca-cert" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:29.980117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.980090 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-socket" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.980198 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.980138 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-certs" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.980309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.980277 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-data" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.980495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.980480 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.980599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.980500 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.980599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.980514 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.980599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.980542 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:29.981954 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.981928 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-token" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:29.982132 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.982107 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-envoy" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:48:29.982226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.982143 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-kube-api-access-fqg6w" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "kube-api-access-fqg6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:29.982277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.982216 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-podinfo" pod "875f34a7-f57b-4c3b-a15b-a6a2df0f816e" (UID: "875f34a7-f57b-4c3b-a15b-a6a2df0f816e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:48:29.990079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.990053 4510 generic.go:356] "Generic (PLEG): container finished" podID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerID="f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad" exitCode=0 Nov 02 21:48:29.990187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.990108 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" Nov 02 21:48:29.990187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.990131 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" event={"ID":"875f34a7-f57b-4c3b-a15b-a6a2df0f816e","Type":"ContainerDied","Data":"f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad"} Nov 02 21:48:29.990300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.990198 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp" event={"ID":"875f34a7-f57b-4c3b-a15b-a6a2df0f816e","Type":"ContainerDied","Data":"760a1ae7fdca12d195668d987b2a9c40e580ad12f3a8b429e4a5f3449bab717a"} Nov 02 21:48:29.990300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.990231 4510 scope.go:122] "RemoveContainer" containerID="f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad" Nov 02 21:48:29.995134 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.995113 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" Nov 02 21:48:29.995235 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:29.995146 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb" event={"ID":"f5e897ad-5975-4ab4-a508-a574208a7aef","Type":"ContainerDied","Data":"9a587e165c4f77acf7d0398a712f14ec3e3e077f554d8914cec1f4845d2179db"} Nov 02 21:48:30.003985 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.003970 4510 scope.go:122] "RemoveContainer" containerID="f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad" Nov 02 21:48:30.004230 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:30.004209 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad\": container with ID starting with f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad not found: ID does not exist" containerID="f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad" Nov 02 21:48:30.004303 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.004241 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad"} err="failed to get container status \"f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad\": rpc error: code = NotFound desc = could not find container \"f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad\": container with ID starting with f5daecd9f3b2fb8009ce18cdbe54fc4fe3a6e461f52c271dc225aeac40be42ad not found: ID does not exist" Nov 02 21:48:30.004303 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.004263 4510 scope.go:122] "RemoveContainer" containerID="cea251b13d9b9e00890eba1a649f5c352af2848d6f29cad5c6efa8740d9540f5" Nov 02 21:48:30.082345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.082312 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqg6w\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-kube-api-access-fqg6w\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:30.082345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.082345 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:30.082345 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.082355 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:30.082604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.082364 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:30.082604 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.082373 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875f34a7-f57b-4c3b-a15b-a6a2df0f816e-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:30.154665 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.154634 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" path="/var/lib/kubelet/pods/c78ab5d8-f481-4ac1-9d2a-5553b735f385/volumes" Nov 02 21:48:30.164011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.163984 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb"] Nov 02 21:48:30.171427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.171364 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4d9ea0-openshift-default-5568bb4fbc-tkbbb"] Nov 02 21:48:30.268435 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.268401 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp"] Nov 02 21:48:30.273720 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:30.273693 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-qtwpp"] Nov 02 21:48:32.156474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:32.156438 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" path="/var/lib/kubelet/pods/875f34a7-f57b-4c3b-a15b-a6a2df0f816e/volumes" Nov 02 21:48:32.157238 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:32.157214 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" path="/var/lib/kubelet/pods/f5e897ad-5975-4ab4-a508-a574208a7aef/volumes" Nov 02 21:48:32.743399 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:32.743366 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf"] Nov 02 21:48:32.743625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:32.743599 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" podUID="09c0c179-138d-476e-8aa0-43102e35f404" containerName="maas-api" containerID="cri-o://2a8680e4debaec44f22ade5c483b4f34e72ed705bb38642abd55dca28591ec29" gracePeriod=30 Nov 02 21:48:33.018400 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.018369 4510 generic.go:356] "Generic (PLEG): container finished" podID="09c0c179-138d-476e-8aa0-43102e35f404" containerID="2a8680e4debaec44f22ade5c483b4f34e72ed705bb38642abd55dca28591ec29" exitCode=0 Nov 02 21:48:33.018583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.018441 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" event={"ID":"09c0c179-138d-476e-8aa0-43102e35f404","Type":"ContainerDied","Data":"2a8680e4debaec44f22ade5c483b4f34e72ed705bb38642abd55dca28591ec29"} Nov 02 21:48:33.018583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.018491 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" event={"ID":"09c0c179-138d-476e-8aa0-43102e35f404","Type":"ContainerDied","Data":"81bbd84873d077c488f14fd23230d7c5ca4e9382097257e12a5cc27dba5fc882"} Nov 02 21:48:33.018583 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.018510 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81bbd84873d077c488f14fd23230d7c5ca4e9382097257e12a5cc27dba5fc882" Nov 02 21:48:33.034704 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.034676 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:48:33.138315 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.138269 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c0c179-138d-476e-8aa0-43102e35f404-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c0c179-138d-476e-8aa0-43102e35f404-openshift-service-ca\") pod \"09c0c179-138d-476e-8aa0-43102e35f404\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " Nov 02 21:48:33.138493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.138393 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-maas-api-tls\") pod \"09c0c179-138d-476e-8aa0-43102e35f404\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " Nov 02 21:48:33.138620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.138489 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-metrics-certs\") pod \"09c0c179-138d-476e-8aa0-43102e35f404\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " Nov 02 21:48:33.138620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.138600 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c0c179-138d-476e-8aa0-43102e35f404-kube-api-access-js2b9\" (UniqueName: \"kubernetes.io/projected/09c0c179-138d-476e-8aa0-43102e35f404-kube-api-access-js2b9\") pod \"09c0c179-138d-476e-8aa0-43102e35f404\" (UID: \"09c0c179-138d-476e-8aa0-43102e35f404\") " Nov 02 21:48:33.138728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.138627 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c0c179-138d-476e-8aa0-43102e35f404-openshift-service-ca" pod "09c0c179-138d-476e-8aa0-43102e35f404" (UID: "09c0c179-138d-476e-8aa0-43102e35f404"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:33.139127 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.139104 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c0c179-138d-476e-8aa0-43102e35f404-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:33.145795 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.140872 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-metrics-certs" pod "09c0c179-138d-476e-8aa0-43102e35f404" (UID: "09c0c179-138d-476e-8aa0-43102e35f404"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:33.145795 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.141111 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c0c179-138d-476e-8aa0-43102e35f404-kube-api-access-js2b9" pod "09c0c179-138d-476e-8aa0-43102e35f404" (UID: "09c0c179-138d-476e-8aa0-43102e35f404"). InnerVolumeSpecName "kube-api-access-js2b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:33.145795 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.141126 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-maas-api-tls" pod "09c0c179-138d-476e-8aa0-43102e35f404" (UID: "09c0c179-138d-476e-8aa0-43102e35f404"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:33.240434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.240400 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2b9\" (UniqueName: \"kubernetes.io/projected/09c0c179-138d-476e-8aa0-43102e35f404-kube-api-access-js2b9\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:33.240434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.240426 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:33.240434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:33.240435 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09c0c179-138d-476e-8aa0-43102e35f404-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:34.025422 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.025386 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf" Nov 02 21:48:34.178227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.178194 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf"] Nov 02 21:48:34.224139 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.224109 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f13b18-8c9879899-sgmrf"] Nov 02 21:48:34.472800 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.472770 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t"] Nov 02 21:48:34.473499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473480 4510 cpu_manager.go:414] "containerMap: removing container" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerName="discovery" Nov 02 21:48:34.473499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473496 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerName="discovery" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473506 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7039890a-43f4-4d63-bc88-1a9d2c54f413" containerName="maas-api" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473512 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7039890a-43f4-4d63-bc88-1a9d2c54f413" containerName="maas-api" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473541 4510 cpu_manager.go:414] "containerMap: removing container" podUID="c74e302b-f55c-4158-a0d3-8a682e6c75eb" containerName="maas-api" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473547 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74e302b-f55c-4158-a0d3-8a682e6c75eb" containerName="maas-api" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473559 4510 cpu_manager.go:414] "containerMap: removing container" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473564 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473580 4510 cpu_manager.go:414] "containerMap: removing container" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473585 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473597 4510 cpu_manager.go:414] "containerMap: removing container" podUID="09c0c179-138d-476e-8aa0-43102e35f404" containerName="maas-api" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473602 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c0c179-138d-476e-8aa0-43102e35f404" containerName="maas-api" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473617 4510 cpu_manager.go:414] "containerMap: removing container" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473621 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473628 4510 cpu_manager.go:414] "containerMap: removing container" podUID="4e4665b8-c27f-44ef-8114-4e692d3dd622" containerName="maas-api" Nov 02 21:48:34.473631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473633 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4665b8-c27f-44ef-8114-4e692d3dd622" containerName="maas-api" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473644 4510 cpu_manager.go:414] "containerMap: removing container" podUID="dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" containerName="maas-api" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473648 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" containerName="maas-api" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473656 4510 cpu_manager.go:414] "containerMap: removing container" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473661 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473673 4510 cpu_manager.go:414] "containerMap: removing container" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473677 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473771 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473782 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78ab5d8-f481-4ac1-9d2a-5553b735f385" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473791 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c0c179-138d-476e-8aa0-43102e35f404" containerName="maas-api" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473799 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4665b8-c27f-44ef-8114-4e692d3dd622" containerName="maas-api" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473805 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa3f2a9-c39b-41f3-9e1f-7c8f0c38f2b2" containerName="maas-api" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473811 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e897ad-5975-4ab4-a508-a574208a7aef" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473819 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="875f34a7-f57b-4c3b-a15b-a6a2df0f816e" containerName="istio-proxy" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473825 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ba9c8d-b0ea-437a-95fd-b4ce24677947" containerName="discovery" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473830 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7039890a-43f4-4d63-bc88-1a9d2c54f413" containerName="maas-api" Nov 02 21:48:34.474047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.473836 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74e302b-f55c-4158-a0d3-8a682e6c75eb" containerName="maas-api" Nov 02 21:48:34.477637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.477621 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.527694 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.527669 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6b983e02-serving-cert\"" Nov 02 21:48:34.531426 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.531401 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t"] Nov 02 21:48:34.558014 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.557979 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-openshift-service-ca\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.558168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.558052 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.558168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.558148 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6r9r\" (UniqueName: \"kubernetes.io/projected/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-kube-api-access-b6r9r\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.558313 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.558293 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-metrics-certs\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.661397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.659513 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-openshift-service-ca\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.661397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.659581 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.661397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.659607 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6r9r\" (UniqueName: \"kubernetes.io/projected/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-kube-api-access-b6r9r\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.661397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.659708 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-metrics-certs\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.661397 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:34.660224 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-serving-cert: secret "maas-api-e2e-mt-6b983e02-serving-cert" not found Nov 02 21:48:34.661397 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:34.660308 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls podName:061f9fe1-ea10-43ca-a32a-8d1aa052efc0 nodeName:}" failed. No retries permitted until 2026-11-02 21:48:35.160288545 +0000 UTC m=+2141.136586321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls") pod "maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" (UID: "061f9fe1-ea10-43ca-a32a-8d1aa052efc0") : secret "maas-api-e2e-mt-6b983e02-serving-cert" not found Nov 02 21:48:34.661397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.660668 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-openshift-service-ca\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.672463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.672435 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-metrics-certs\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:34.713163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:34.713142 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6r9r\" (UniqueName: \"kubernetes.io/projected/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-kube-api-access-b6r9r\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:35.167042 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:35.167008 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:35.169483 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:35.169460 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls\") pod \"maas-api-e2e-mt-6b983e02-545d64b797-9pl7t\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:35.391714 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:35.391683 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:35.765338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:35.765252 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t"] Nov 02 21:48:35.771722 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:48:35.771694 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061f9fe1_ea10_43ca_a32a_8d1aa052efc0.slice/crio-347f282792784a13eb14fe1b2377b153810b3f526d508806e9ccf0171c2018b3 WatchSource:0}: Error finding container 347f282792784a13eb14fe1b2377b153810b3f526d508806e9ccf0171c2018b3: Status 404 returned error can't find the container with id 347f282792784a13eb14fe1b2377b153810b3f526d508806e9ccf0171c2018b3 Nov 02 21:48:35.773276 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:35.773252 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:48:36.045204 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:36.045127 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" event={"ID":"061f9fe1-ea10-43ca-a32a-8d1aa052efc0","Type":"ContainerStarted","Data":"347f282792784a13eb14fe1b2377b153810b3f526d508806e9ccf0171c2018b3"} Nov 02 21:48:36.162806 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:36.162773 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c0c179-138d-476e-8aa0-43102e35f404" path="/var/lib/kubelet/pods/09c0c179-138d-476e-8aa0-43102e35f404/volumes" Nov 02 21:48:37.054391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:37.054305 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" event={"ID":"061f9fe1-ea10-43ca-a32a-8d1aa052efc0","Type":"ContainerStarted","Data":"23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51"} Nov 02 21:48:37.054391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:37.054365 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:37.161792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:37.161725 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" podStartSLOduration=2.209979126 podStartE2EDuration="3.161712095s" totalImagesPullingTime="951.732969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:48:37.160348562 +0000 UTC m=+2143.136646337" watchObservedRunningTime="2026-11-02 21:48:37.161712095 +0000 UTC m=+2143.138009870" Nov 02 21:48:39.147363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.147329 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz"] Nov 02 21:48:39.147876 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.147541 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" podUID="a62e2eb2-f380-4c14-8ddb-42a946d85202" containerName="maas-api" containerID="cri-o://c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3" gracePeriod=30 Nov 02 21:48:39.433017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.432992 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:48:39.452070 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.452044 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls\") pod \"a62e2eb2-f380-4c14-8ddb-42a946d85202\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " Nov 02 21:48:39.452197 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.452171 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-metrics-certs\") pod \"a62e2eb2-f380-4c14-8ddb-42a946d85202\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " Nov 02 21:48:39.452254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.452203 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62e2eb2-f380-4c14-8ddb-42a946d85202-kube-api-access-q62pw\" (UniqueName: \"kubernetes.io/projected/a62e2eb2-f380-4c14-8ddb-42a946d85202-kube-api-access-q62pw\") pod \"a62e2eb2-f380-4c14-8ddb-42a946d85202\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " Nov 02 21:48:39.452254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.452224 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62e2eb2-f380-4c14-8ddb-42a946d85202-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e2eb2-f380-4c14-8ddb-42a946d85202-openshift-service-ca\") pod \"a62e2eb2-f380-4c14-8ddb-42a946d85202\" (UID: \"a62e2eb2-f380-4c14-8ddb-42a946d85202\") " Nov 02 21:48:39.452846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.452691 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62e2eb2-f380-4c14-8ddb-42a946d85202-openshift-service-ca" pod "a62e2eb2-f380-4c14-8ddb-42a946d85202" (UID: "a62e2eb2-f380-4c14-8ddb-42a946d85202"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:48:39.452846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.452816 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62e2eb2-f380-4c14-8ddb-42a946d85202-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:39.454755 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.454614 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-metrics-certs" pod "a62e2eb2-f380-4c14-8ddb-42a946d85202" (UID: "a62e2eb2-f380-4c14-8ddb-42a946d85202"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:39.454755 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.454695 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62e2eb2-f380-4c14-8ddb-42a946d85202-kube-api-access-q62pw" pod "a62e2eb2-f380-4c14-8ddb-42a946d85202" (UID: "a62e2eb2-f380-4c14-8ddb-42a946d85202"). InnerVolumeSpecName "kube-api-access-q62pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:48:39.454901 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.454852 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls" pod "a62e2eb2-f380-4c14-8ddb-42a946d85202" (UID: "a62e2eb2-f380-4c14-8ddb-42a946d85202"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:48:39.554079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.554042 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62pw\" (UniqueName: \"kubernetes.io/projected/a62e2eb2-f380-4c14-8ddb-42a946d85202-kube-api-access-q62pw\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:39.554079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.554081 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:39.554332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:39.554096 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a62e2eb2-f380-4c14-8ddb-42a946d85202-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:48:40.077266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.077235 4510 generic.go:356] "Generic (PLEG): container finished" podID="a62e2eb2-f380-4c14-8ddb-42a946d85202" containerID="c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3" exitCode=0 Nov 02 21:48:40.077447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.077293 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" Nov 02 21:48:40.077447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.077324 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" event={"ID":"a62e2eb2-f380-4c14-8ddb-42a946d85202","Type":"ContainerDied","Data":"c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3"} Nov 02 21:48:40.077447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.077385 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz" event={"ID":"a62e2eb2-f380-4c14-8ddb-42a946d85202","Type":"ContainerDied","Data":"8625223a20a7cdb1c86165031a020ca317b3ad40f66dd89a52e2ad6ba02d15ad"} Nov 02 21:48:40.077447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.077408 4510 scope.go:122] "RemoveContainer" containerID="c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3" Nov 02 21:48:40.089354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.089335 4510 scope.go:122] "RemoveContainer" containerID="c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3" Nov 02 21:48:40.089654 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:40.089634 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3\": container with ID starting with c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3 not found: ID does not exist" containerID="c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3" Nov 02 21:48:40.089734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.089660 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3"} err="failed to get container status \"c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3\": rpc error: code = NotFound desc = could not find container \"c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3\": container with ID starting with c02bab108910c01149ca216188b7fd50d9cde0c2e878a04d973f4f5dc35e6fc3 not found: ID does not exist" Nov 02 21:48:40.234832 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.234796 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz"] Nov 02 21:48:40.281268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:40.281229 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7107c-67b687b789-r2whz"] Nov 02 21:48:42.155868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:42.155830 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62e2eb2-f380-4c14-8ddb-42a946d85202" path="/var/lib/kubelet/pods/a62e2eb2-f380-4c14-8ddb-42a946d85202/volumes" Nov 02 21:48:43.069368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:48:43.069337 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:48:45.548261 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:48:45.548215 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:49:02.986018 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:02.985981 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc"] Nov 02 21:49:03.087919 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:03.087885 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64"] Nov 02 21:49:05.737652 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:05.737622 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:49:07.986849 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:07.986809 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" podUID="b096eae1-e575-472d-aba6-50da1218fb46" containerName="payload-processing" containerID="cri-o://79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707" gracePeriod=25 Nov 02 21:49:08.088279 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.088246 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" podUID="7c62a903-b11e-4011-8e2b-7d34022876c5" containerName="payload-pre-processing" containerID="cri-o://c95840ab51ecf87d68ce55bb00d52b6d9c5c88f1bee5d69d863a0824a2e4d512" gracePeriod=25 Nov 02 21:49:08.267249 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.267223 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:49:08.312161 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.312121 4510 generic.go:356] "Generic (PLEG): container finished" podID="b096eae1-e575-472d-aba6-50da1218fb46" containerID="79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707" exitCode=0 Nov 02 21:49:08.312326 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.312307 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" event={"ID":"b096eae1-e575-472d-aba6-50da1218fb46","Type":"ContainerDied","Data":"79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707"} Nov 02 21:49:08.312395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.312359 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" event={"ID":"b096eae1-e575-472d-aba6-50da1218fb46","Type":"ContainerDied","Data":"293050c79465819322b29269790878db3e004a6683cb2b69d4ef4e12a35654a3"} Nov 02 21:49:08.312395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.312390 4510 scope.go:122] "RemoveContainer" containerID="79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707" Nov 02 21:49:08.312578 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.312561 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc" Nov 02 21:49:08.314842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.314816 4510 generic.go:356] "Generic (PLEG): container finished" podID="7c62a903-b11e-4011-8e2b-7d34022876c5" containerID="c95840ab51ecf87d68ce55bb00d52b6d9c5c88f1bee5d69d863a0824a2e4d512" exitCode=0 Nov 02 21:49:08.315005 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.314985 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" event={"ID":"7c62a903-b11e-4011-8e2b-7d34022876c5","Type":"ContainerDied","Data":"c95840ab51ecf87d68ce55bb00d52b6d9c5c88f1bee5d69d863a0824a2e4d512"} Nov 02 21:49:08.325051 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.325029 4510 scope.go:122] "RemoveContainer" containerID="79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707" Nov 02 21:49:08.325398 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:08.325368 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707\": container with ID starting with 79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707 not found: ID does not exist" containerID="79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707" Nov 02 21:49:08.325575 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.325547 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707"} err="failed to get container status \"79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707\": rpc error: code = NotFound desc = could not find container \"79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707\": container with ID starting with 79bf81139124b455d0b84a9f3dcb533b01a2297c755652325943eb26af8eb707 not found: ID does not exist" Nov 02 21:49:08.353280 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.353261 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b096eae1-e575-472d-aba6-50da1218fb46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b096eae1-e575-472d-aba6-50da1218fb46-plugins-config-volume\") pod \"b096eae1-e575-472d-aba6-50da1218fb46\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " Nov 02 21:49:08.353375 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.353347 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096eae1-e575-472d-aba6-50da1218fb46-kube-api-access-6gcq4\" (UniqueName: \"kubernetes.io/projected/b096eae1-e575-472d-aba6-50da1218fb46-kube-api-access-6gcq4\") pod \"b096eae1-e575-472d-aba6-50da1218fb46\" (UID: \"b096eae1-e575-472d-aba6-50da1218fb46\") " Nov 02 21:49:08.353617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.353598 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b096eae1-e575-472d-aba6-50da1218fb46-plugins-config-volume" pod "b096eae1-e575-472d-aba6-50da1218fb46" (UID: "b096eae1-e575-472d-aba6-50da1218fb46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:08.353901 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.353882 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b096eae1-e575-472d-aba6-50da1218fb46-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:08.355169 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.355150 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:49:08.355328 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.355307 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096eae1-e575-472d-aba6-50da1218fb46-kube-api-access-6gcq4" pod "b096eae1-e575-472d-aba6-50da1218fb46" (UID: "b096eae1-e575-472d-aba6-50da1218fb46"). InnerVolumeSpecName "kube-api-access-6gcq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:08.454679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.454646 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c62a903-b11e-4011-8e2b-7d34022876c5-kube-api-access-skkvc\" (UniqueName: \"kubernetes.io/projected/7c62a903-b11e-4011-8e2b-7d34022876c5-kube-api-access-skkvc\") pod \"7c62a903-b11e-4011-8e2b-7d34022876c5\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " Nov 02 21:49:08.454844 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.454756 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c62a903-b11e-4011-8e2b-7d34022876c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c62a903-b11e-4011-8e2b-7d34022876c5-plugins-config-volume\") pod \"7c62a903-b11e-4011-8e2b-7d34022876c5\" (UID: \"7c62a903-b11e-4011-8e2b-7d34022876c5\") " Nov 02 21:49:08.455125 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.455095 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c62a903-b11e-4011-8e2b-7d34022876c5-plugins-config-volume" pod "7c62a903-b11e-4011-8e2b-7d34022876c5" (UID: "7c62a903-b11e-4011-8e2b-7d34022876c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:08.455310 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.455287 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c62a903-b11e-4011-8e2b-7d34022876c5-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:08.455310 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.455316 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcq4\" (UniqueName: \"kubernetes.io/projected/b096eae1-e575-472d-aba6-50da1218fb46-kube-api-access-6gcq4\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:08.456809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.456787 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c62a903-b11e-4011-8e2b-7d34022876c5-kube-api-access-skkvc" pod "7c62a903-b11e-4011-8e2b-7d34022876c5" (UID: "7c62a903-b11e-4011-8e2b-7d34022876c5"). InnerVolumeSpecName "kube-api-access-skkvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:08.556779 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.556708 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skkvc\" (UniqueName: \"kubernetes.io/projected/7c62a903-b11e-4011-8e2b-7d34022876c5-kube-api-access-skkvc\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:08.725246 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.725214 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc"] Nov 02 21:49:08.729443 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:08.729413 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-32de96-5ff6bbb5ff-rqwzc"] Nov 02 21:49:09.328833 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:09.328755 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" Nov 02 21:49:09.329266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:09.328761 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64" event={"ID":"7c62a903-b11e-4011-8e2b-7d34022876c5","Type":"ContainerDied","Data":"c7039c5a5acfe8138e0e9d7efafb8faf986be51ad79eb88be99d652fb398b114"} Nov 02 21:49:09.329266 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:09.328883 4510 scope.go:122] "RemoveContainer" containerID="c95840ab51ecf87d68ce55bb00d52b6d9c5c88f1bee5d69d863a0824a2e4d512" Nov 02 21:49:09.482029 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:09.481996 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64"] Nov 02 21:49:09.528101 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:09.528074 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-32de96-5dc88f798c-hxh64"] Nov 02 21:49:10.158023 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:10.157984 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c62a903-b11e-4011-8e2b-7d34022876c5" path="/var/lib/kubelet/pods/7c62a903-b11e-4011-8e2b-7d34022876c5/volumes" Nov 02 21:49:10.158793 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:10.158761 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b096eae1-e575-472d-aba6-50da1218fb46" path="/var/lib/kubelet/pods/b096eae1-e575-472d-aba6-50da1218fb46/volumes" Nov 02 21:49:15.530924 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:15.530893 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:49:23.823656 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:23.823621 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9"] Nov 02 21:49:23.824166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:23.823856 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" podUID="93e34ccd-1b03-4b2f-8a1f-3408627570da" containerName="maas-api" containerID="cri-o://e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445" gracePeriod=30 Nov 02 21:49:24.105123 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.105097 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:49:24.249594 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.249564 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls\") pod \"93e34ccd-1b03-4b2f-8a1f-3408627570da\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " Nov 02 21:49:24.249756 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.249692 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-metrics-certs\") pod \"93e34ccd-1b03-4b2f-8a1f-3408627570da\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " Nov 02 21:49:24.249908 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.249885 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e34ccd-1b03-4b2f-8a1f-3408627570da-kube-api-access-bn827\" (UniqueName: \"kubernetes.io/projected/93e34ccd-1b03-4b2f-8a1f-3408627570da-kube-api-access-bn827\") pod \"93e34ccd-1b03-4b2f-8a1f-3408627570da\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " Nov 02 21:49:24.249983 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.249968 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e34ccd-1b03-4b2f-8a1f-3408627570da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e34ccd-1b03-4b2f-8a1f-3408627570da-openshift-service-ca\") pod \"93e34ccd-1b03-4b2f-8a1f-3408627570da\" (UID: \"93e34ccd-1b03-4b2f-8a1f-3408627570da\") " Nov 02 21:49:24.250574 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.250502 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e34ccd-1b03-4b2f-8a1f-3408627570da-openshift-service-ca" pod "93e34ccd-1b03-4b2f-8a1f-3408627570da" (UID: "93e34ccd-1b03-4b2f-8a1f-3408627570da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:24.250787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.250765 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93e34ccd-1b03-4b2f-8a1f-3408627570da-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:24.252719 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.252682 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls" pod "93e34ccd-1b03-4b2f-8a1f-3408627570da" (UID: "93e34ccd-1b03-4b2f-8a1f-3408627570da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:24.253066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.253028 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e34ccd-1b03-4b2f-8a1f-3408627570da-kube-api-access-bn827" pod "93e34ccd-1b03-4b2f-8a1f-3408627570da" (UID: "93e34ccd-1b03-4b2f-8a1f-3408627570da"). InnerVolumeSpecName "kube-api-access-bn827". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:24.253336 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.253317 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-metrics-certs" pod "93e34ccd-1b03-4b2f-8a1f-3408627570da" (UID: "93e34ccd-1b03-4b2f-8a1f-3408627570da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:24.352825 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.352737 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:24.352825 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.352767 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e34ccd-1b03-4b2f-8a1f-3408627570da-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:24.352825 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.352779 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn827\" (UniqueName: \"kubernetes.io/projected/93e34ccd-1b03-4b2f-8a1f-3408627570da-kube-api-access-bn827\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:24.424053 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.424018 4510 generic.go:356] "Generic (PLEG): container finished" podID="93e34ccd-1b03-4b2f-8a1f-3408627570da" containerID="e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445" exitCode=0 Nov 02 21:49:24.424239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.424070 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" Nov 02 21:49:24.424239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.424106 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" event={"ID":"93e34ccd-1b03-4b2f-8a1f-3408627570da","Type":"ContainerDied","Data":"e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445"} Nov 02 21:49:24.424239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.424173 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" event={"ID":"93e34ccd-1b03-4b2f-8a1f-3408627570da","Type":"ContainerDied","Data":"50ae5bd0c1ecb00f9edb046dcc5db648ebac44ac68c7a6cf8a9c5176338b49fa"} Nov 02 21:49:24.424239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.424205 4510 scope.go:122] "RemoveContainer" containerID="e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445" Nov 02 21:49:24.435707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.435691 4510 scope.go:122] "RemoveContainer" containerID="e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445" Nov 02 21:49:24.435989 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:24.435969 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445\": container with ID starting with e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445 not found: ID does not exist" containerID="e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445" Nov 02 21:49:24.436048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.435998 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445"} err="failed to get container status \"e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445\": rpc error: code = NotFound desc = could not find container \"e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445\": container with ID starting with e133e20974507a2e2b5ea4fd0e2f74ed3bca458a839c79cf0c43ffa2e98eb445 not found: ID does not exist" Nov 02 21:49:24.577966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.577937 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9"] Nov 02 21:49:24.580955 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:24.580929 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9"] Nov 02 21:49:26.155162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:26.155130 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e34ccd-1b03-4b2f-8a1f-3408627570da" path="/var/lib/kubelet/pods/93e34ccd-1b03-4b2f-8a1f-3408627570da/volumes" Nov 02 21:49:27.099599 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.099561 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-5451a5-68659c88c4-ddjl9" podUID="93e34ccd-1b03-4b2f-8a1f-3408627570da" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:49:27.335506 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.335474 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg"] Nov 02 21:49:27.336186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336170 4510 cpu_manager.go:414] "containerMap: removing container" podUID="93e34ccd-1b03-4b2f-8a1f-3408627570da" containerName="maas-api" Nov 02 21:49:27.336186 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336187 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e34ccd-1b03-4b2f-8a1f-3408627570da" containerName="maas-api" Nov 02 21:49:27.336286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336220 4510 cpu_manager.go:414] "containerMap: removing container" podUID="a62e2eb2-f380-4c14-8ddb-42a946d85202" containerName="maas-api" Nov 02 21:49:27.336286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336225 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62e2eb2-f380-4c14-8ddb-42a946d85202" containerName="maas-api" Nov 02 21:49:27.336286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336233 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7c62a903-b11e-4011-8e2b-7d34022876c5" containerName="payload-pre-processing" Nov 02 21:49:27.336286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336238 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c62a903-b11e-4011-8e2b-7d34022876c5" containerName="payload-pre-processing" Nov 02 21:49:27.336286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336264 4510 cpu_manager.go:414] "containerMap: removing container" podUID="b096eae1-e575-472d-aba6-50da1218fb46" containerName="payload-processing" Nov 02 21:49:27.336286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336269 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b096eae1-e575-472d-aba6-50da1218fb46" containerName="payload-processing" Nov 02 21:49:27.336495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336389 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e34ccd-1b03-4b2f-8a1f-3408627570da" containerName="maas-api" Nov 02 21:49:27.336495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336400 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6d30ce-9a52-400e-a7c3-0937edd9f564" containerName="istio-proxy" Nov 02 21:49:27.336495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336407 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62e2eb2-f380-4c14-8ddb-42a946d85202" containerName="maas-api" Nov 02 21:49:27.336495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336412 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c62a903-b11e-4011-8e2b-7d34022876c5" containerName="payload-pre-processing" Nov 02 21:49:27.336495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.336420 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b096eae1-e575-472d-aba6-50da1218fb46" containerName="payload-processing" Nov 02 21:49:27.341068 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.341050 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.389477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.389449 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b983e02-openshift-default-dockercfg-xbqwp\"" Nov 02 21:49:27.397770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.397733 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.397899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.397843 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.398010 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.397994 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.398069 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.398041 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nhd\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-kube-api-access-q9nhd\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.398165 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.398145 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.398205 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.398171 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.398253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.398203 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.398253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.398232 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.398348 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.398333 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.435473 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.435442 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg"] Nov 02 21:49:27.499855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.499818 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.499855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.499859 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.499903 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.499940 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500094 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.499982 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500110 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500158 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500222 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500308 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500372 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500432 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nhd\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-kube-api-access-q9nhd\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500560 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500612 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.500982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.500759 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.502598 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.502581 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.502655 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.502636 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.553143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.553106 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nhd\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-kube-api-access-q9nhd\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.553319 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.553154 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:27.655226 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:27.655144 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:28.092571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.092476 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg"] Nov 02 21:49:28.098752 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:28.098726 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84072bc7_dcbf_4c58_93aa_8fa81ebe31e0.slice/crio-d2a56d1146150fdfb03560bcd7d9dbc812a58e6e2efe95c90bdef2e58d4bc461 WatchSource:0}: Error finding container d2a56d1146150fdfb03560bcd7d9dbc812a58e6e2efe95c90bdef2e58d4bc461: Status 404 returned error can't find the container with id d2a56d1146150fdfb03560bcd7d9dbc812a58e6e2efe95c90bdef2e58d4bc461 Nov 02 21:49:28.101405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.101371 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:28.101492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.101438 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:28.101492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.101468 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:28.453981 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.453943 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" event={"ID":"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0","Type":"ContainerStarted","Data":"c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0"} Nov 02 21:49:28.454400 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.453997 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" event={"ID":"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0","Type":"ContainerStarted","Data":"d2a56d1146150fdfb03560bcd7d9dbc812a58e6e2efe95c90bdef2e58d4bc461"} Nov 02 21:49:28.564970 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.564916 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" podStartSLOduration=1.564901553 podStartE2EDuration="1.564901553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:28.56189532 +0000 UTC m=+2194.538193095" watchObservedRunningTime="2026-11-02 21:49:28.564901553 +0000 UTC m=+2194.541199327" Nov 02 21:49:28.655533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:28.655483 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:29.657924 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:29.657887 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:49:29.658338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:29.657953 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 21:49:29.672869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:29.672840 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p"] Nov 02 21:49:29.673082 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:29.673056 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" podUID="a9801076-2ad0-4395-b7e0-a36d63fd4852" containerName="maas-api" containerID="cri-o://0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb" gracePeriod=30 Nov 02 21:49:30.009994 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.009964 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:49:30.067763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.067732 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls\") pod \"a9801076-2ad0-4395-b7e0-a36d63fd4852\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " Nov 02 21:49:30.067947 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.067822 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9801076-2ad0-4395-b7e0-a36d63fd4852-kube-api-access-r5gxx\" (UniqueName: \"kubernetes.io/projected/a9801076-2ad0-4395-b7e0-a36d63fd4852-kube-api-access-r5gxx\") pod \"a9801076-2ad0-4395-b7e0-a36d63fd4852\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " Nov 02 21:49:30.067947 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.067924 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9801076-2ad0-4395-b7e0-a36d63fd4852-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9801076-2ad0-4395-b7e0-a36d63fd4852-openshift-service-ca\") pod \"a9801076-2ad0-4395-b7e0-a36d63fd4852\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " Nov 02 21:49:30.068029 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.068006 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-metrics-certs\") pod \"a9801076-2ad0-4395-b7e0-a36d63fd4852\" (UID: \"a9801076-2ad0-4395-b7e0-a36d63fd4852\") " Nov 02 21:49:30.068229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.068203 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9801076-2ad0-4395-b7e0-a36d63fd4852-openshift-service-ca" pod "a9801076-2ad0-4395-b7e0-a36d63fd4852" (UID: "a9801076-2ad0-4395-b7e0-a36d63fd4852"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:30.068588 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.068569 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9801076-2ad0-4395-b7e0-a36d63fd4852-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:30.070031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.069996 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-metrics-certs" pod "a9801076-2ad0-4395-b7e0-a36d63fd4852" (UID: "a9801076-2ad0-4395-b7e0-a36d63fd4852"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:30.070451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.070424 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls" pod "a9801076-2ad0-4395-b7e0-a36d63fd4852" (UID: "a9801076-2ad0-4395-b7e0-a36d63fd4852"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:30.070542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.070452 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9801076-2ad0-4395-b7e0-a36d63fd4852-kube-api-access-r5gxx" pod "a9801076-2ad0-4395-b7e0-a36d63fd4852" (UID: "a9801076-2ad0-4395-b7e0-a36d63fd4852"). InnerVolumeSpecName "kube-api-access-r5gxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:30.173537 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.170369 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5gxx\" (UniqueName: \"kubernetes.io/projected/a9801076-2ad0-4395-b7e0-a36d63fd4852-kube-api-access-r5gxx\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:30.173537 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.170397 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:30.173537 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.170407 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9801076-2ad0-4395-b7e0-a36d63fd4852-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:30.468906 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.468871 4510 generic.go:356] "Generic (PLEG): container finished" podID="a9801076-2ad0-4395-b7e0-a36d63fd4852" containerID="0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb" exitCode=0 Nov 02 21:49:30.469065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.468924 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" Nov 02 21:49:30.469065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.468948 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" event={"ID":"a9801076-2ad0-4395-b7e0-a36d63fd4852","Type":"ContainerDied","Data":"0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb"} Nov 02 21:49:30.469065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.469009 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p" event={"ID":"a9801076-2ad0-4395-b7e0-a36d63fd4852","Type":"ContainerDied","Data":"ed99c0f2d793e849863151d7d0896eab8a8a10a7c8d023238679ac945998bfcd"} Nov 02 21:49:30.469065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.469034 4510 scope.go:122] "RemoveContainer" containerID="0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb" Nov 02 21:49:30.479434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.479411 4510 scope.go:122] "RemoveContainer" containerID="0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb" Nov 02 21:49:30.479703 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:30.479686 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb\": container with ID starting with 0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb not found: ID does not exist" containerID="0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb" Nov 02 21:49:30.479772 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.479711 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb"} err="failed to get container status \"0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb\": rpc error: code = NotFound desc = could not find container \"0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb\": container with ID starting with 0cb70bd053e451e6822d1ad8c5d744edef60b9a66e3c77a53e361a6e0fc9dcdb not found: ID does not exist" Nov 02 21:49:30.622447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.622411 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p"] Nov 02 21:49:30.645543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.645070 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-66c72e-8494d96d75-85s7p"] Nov 02 21:49:30.656231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.656199 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:49:30.656397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:30.656263 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 21:49:31.502669 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.502635 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl"] Nov 02 21:49:31.503420 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.503400 4510 cpu_manager.go:414] "containerMap: removing container" podUID="a9801076-2ad0-4395-b7e0-a36d63fd4852" containerName="maas-api" Nov 02 21:49:31.503478 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.503421 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9801076-2ad0-4395-b7e0-a36d63fd4852" containerName="maas-api" Nov 02 21:49:31.503576 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.503566 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9801076-2ad0-4395-b7e0-a36d63fd4852" containerName="maas-api" Nov 02 21:49:31.508348 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.508330 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.556946 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.556917 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7cc1b1-serving-cert\"" Nov 02 21:49:31.587040 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.586382 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-metrics-certs\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.587040 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.586438 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s92fk\" (UniqueName: \"kubernetes.io/projected/40705686-3e8f-4f9c-8bc4-a647a83827b3-kube-api-access-s92fk\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.587040 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.586609 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.587040 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.586715 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40705686-3e8f-4f9c-8bc4-a647a83827b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.599263 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.599232 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl"] Nov 02 21:49:31.656231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.656196 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:49:31.656395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.656260 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 21:49:31.688175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.688145 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.688317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.688254 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40705686-3e8f-4f9c-8bc4-a647a83827b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.688317 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:31.688300 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-serving-cert: secret "maas-api-e2e-inf-a-7cc1b1-serving-cert" not found Nov 02 21:49:31.688396 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:31.688369 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls podName:40705686-3e8f-4f9c-8bc4-a647a83827b3 nodeName:}" failed. No retries permitted until 2026-11-02 21:49:32.188348484 +0000 UTC m=+2198.164646240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls") pod "maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" (UID: "40705686-3e8f-4f9c-8bc4-a647a83827b3") : secret "maas-api-e2e-inf-a-7cc1b1-serving-cert" not found Nov 02 21:49:31.688450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.688396 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-metrics-certs\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.688450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.688426 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s92fk\" (UniqueName: \"kubernetes.io/projected/40705686-3e8f-4f9c-8bc4-a647a83827b3-kube-api-access-s92fk\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.688888 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.688870 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40705686-3e8f-4f9c-8bc4-a647a83827b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.690805 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.690784 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-metrics-certs\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:31.755303 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:31.755229 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s92fk\" (UniqueName: \"kubernetes.io/projected/40705686-3e8f-4f9c-8bc4-a647a83827b3-kube-api-access-s92fk\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:32.158542 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.158495 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9801076-2ad0-4395-b7e0-a36d63fd4852" path="/var/lib/kubelet/pods/a9801076-2ad0-4395-b7e0-a36d63fd4852/volumes" Nov 02 21:49:32.195930 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.195886 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:32.196109 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:32.196024 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-serving-cert: secret "maas-api-e2e-inf-a-7cc1b1-serving-cert" not found Nov 02 21:49:32.196109 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:32.196085 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls podName:40705686-3e8f-4f9c-8bc4-a647a83827b3 nodeName:}" failed. No retries permitted until 2026-11-02 21:49:33.196072226 +0000 UTC m=+2199.172369980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls") pod "maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" (UID: "40705686-3e8f-4f9c-8bc4-a647a83827b3") : secret "maas-api-e2e-inf-a-7cc1b1-serving-cert" not found Nov 02 21:49:32.655457 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.655422 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:49:32.655850 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.655471 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 21:49:32.851269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.851144 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb"] Nov 02 21:49:32.851481 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.851450 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" containerID="cri-o://94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15" gracePeriod=30 Nov 02 21:49:32.854625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.854584 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9"] Nov 02 21:49:32.854878 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.854828 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" containerID="cri-o://53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87" gracePeriod=30 Nov 02 21:49:32.858767 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.858745 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr"] Nov 02 21:49:32.858969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:32.858937 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" containerID="cri-o://7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd" gracePeriod=30 Nov 02 21:49:33.209355 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:33.209326 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:33.209503 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:33.209456 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-serving-cert: secret "maas-api-e2e-inf-a-7cc1b1-serving-cert" not found Nov 02 21:49:33.209573 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:33.209509 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls podName:40705686-3e8f-4f9c-8bc4-a647a83827b3 nodeName:}" failed. No retries permitted until 2026-11-02 21:49:35.209495468 +0000 UTC m=+2201.185793226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls") pod "maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" (UID: "40705686-3e8f-4f9c-8bc4-a647a83827b3") : secret "maas-api-e2e-inf-a-7cc1b1-serving-cert" not found Nov 02 21:49:33.656058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:33.656018 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 02 21:49:33.656437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:33.656082 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 02 21:49:34.716653 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:34.716619 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:34.717130 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:34.716930 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:34.717783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:34.717741 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:49:35.262874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:35.262842 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:35.265360 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:35.265334 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:35.422203 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:35.422171 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:35.797285 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:35.797255 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl"] Nov 02 21:49:35.805570 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:35.805503 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40705686_3e8f_4f9c_8bc4_a647a83827b3.slice/crio-d3023a76837d109270f20e8e8e846a3704de3c58aeeeecd2a02daf750a1c8ed2 WatchSource:0}: Error finding container d3023a76837d109270f20e8e8e846a3704de3c58aeeeecd2a02daf750a1c8ed2: Status 404 returned error can't find the container with id d3023a76837d109270f20e8e8e846a3704de3c58aeeeecd2a02daf750a1c8ed2 Nov 02 21:49:36.515900 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:36.515861 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" event={"ID":"40705686-3e8f-4f9c-8bc4-a647a83827b3","Type":"ContainerStarted","Data":"d3023a76837d109270f20e8e8e846a3704de3c58aeeeecd2a02daf750a1c8ed2"} Nov 02 21:49:37.523764 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:37.523727 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" event={"ID":"40705686-3e8f-4f9c-8bc4-a647a83827b3","Type":"ContainerStarted","Data":"9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb"} Nov 02 21:49:37.524133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:37.523872 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:37.633933 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:37.633879 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" podStartSLOduration=5.749396883 podStartE2EDuration="6.633864178s" totalImagesPullingTime="884.467295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:37.632213896 +0000 UTC m=+2203.608511671" watchObservedRunningTime="2026-11-02 21:49:37.633864178 +0000 UTC m=+2203.610161967" Nov 02 21:49:38.166407 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.162986 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:49:38.178888 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.178864 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:49:38.184643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.184623 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:49:38.303905 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.303873 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-envoy\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.304061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.303927 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-socket\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.303980 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-token\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-token\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304016 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0192173-9c63-4856-802e-48190aae0942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0192173-9c63-4856-802e-48190aae0942-istiod-ca-cert\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.304061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304052 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-credential-socket\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304108 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-kube-api-access-kch2q\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-kube-api-access-kch2q\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.304251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304144 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-certs\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.304251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304176 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-data\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304226 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-kube-api-access-cwlbm\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-kube-api-access-cwlbm\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304270 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-socket" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.304477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304287 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c2a84c-9317-4577-953a-2191b2874cc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c2a84c-9317-4577-953a-2191b2874cc9-istiod-ca-cert\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.304477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304314 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-istio-token\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-istio-token\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.304477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304338 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-credential-socket" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.304477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304368 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83c2a84c-9317-4577-953a-2191b2874cc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c2a84c-9317-4577-953a-2191b2874cc9-istio-podinfo\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.304477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304399 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-podinfo\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304439 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-data\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304487 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0192173-9c63-4856-802e-48190aae0942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0192173-9c63-4856-802e-48190aae0942-istio-podinfo\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304502 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-certs" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304571 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-credential-socket\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304665 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-istio-token\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-istio-token\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304703 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c2a84c-9317-4577-953a-2191b2874cc9-istiod-ca-cert" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304728 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-certs\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304781 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-credential-socket\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304795 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-credential-socket" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304823 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-envoy\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304827 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0192173-9c63-4856-802e-48190aae0942-istiod-ca-cert" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:38.304896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.304873 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-data" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.305468 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305297 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-certs" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.305576 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305536 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-certs\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.305647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305613 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3f2691-eb8c-413b-b17c-5938212aeec1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3f2691-eb8c-413b-b17c-5938212aeec1-istiod-ca-cert\") pod \"cf3f2691-eb8c-413b-b17c-5938212aeec1\" (UID: \"cf3f2691-eb8c-413b-b17c-5938212aeec1\") " Nov 02 21:49:38.305647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305535 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-credential-socket" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.305647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305565 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-data" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.305827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305684 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-socket\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.305827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305717 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-kube-api-access-d6vv6\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-kube-api-access-d6vv6\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.305827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305764 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-socket\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.305827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305770 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-certs" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.305827 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305799 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-envoy\") pod \"83c2a84c-9317-4577-953a-2191b2874cc9\" (UID: \"83c2a84c-9317-4577-953a-2191b2874cc9\") " Nov 02 21:49:38.306096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.305872 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-data\") pod \"a0192173-9c63-4856-802e-48190aae0942\" (UID: \"a0192173-9c63-4856-802e-48190aae0942\") " Nov 02 21:49:38.306096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.306022 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3f2691-eb8c-413b-b17c-5938212aeec1-istiod-ca-cert" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:38.306941 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.306825 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-socket" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.307054 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.306973 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-token" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.307054 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307019 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-socket" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.307254 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307212 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-data" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.307331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307256 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0192173-9c63-4856-802e-48190aae0942-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307281 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307296 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307310 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307325 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c2a84c-9317-4577-953a-2191b2874cc9-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307337 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307351 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307364 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307378 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307391 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307405 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3f2691-eb8c-413b-b17c-5938212aeec1-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307418 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307431 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307445 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.307643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307459 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.308192 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.307842 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-kube-api-access-cwlbm" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "kube-api-access-cwlbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.308395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.308373 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-istio-token" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.308780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.308746 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-kube-api-access-kch2q" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "kube-api-access-kch2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.308780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.308756 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-podinfo" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:38.308961 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.308942 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-envoy" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.309063 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.309033 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-kube-api-access-d6vv6" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "kube-api-access-d6vv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.309180 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.309132 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0192173-9c63-4856-802e-48190aae0942-istio-podinfo" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:38.309592 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.309574 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-envoy" pod "cf3f2691-eb8c-413b-b17c-5938212aeec1" (UID: "cf3f2691-eb8c-413b-b17c-5938212aeec1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.309682 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.309666 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c2a84c-9317-4577-953a-2191b2874cc9-istio-podinfo" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:49:38.309842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.309816 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-envoy" pod "83c2a84c-9317-4577-953a-2191b2874cc9" (UID: "83c2a84c-9317-4577-953a-2191b2874cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:49:38.310055 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.310034 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-istio-token" pod "a0192173-9c63-4856-802e-48190aae0942" (UID: "a0192173-9c63-4856-802e-48190aae0942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:38.408917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408879 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kch2q\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-kube-api-access-kch2q\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.408917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408910 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwlbm\" (UniqueName: \"kubernetes.io/projected/cf3f2691-eb8c-413b-b17c-5938212aeec1-kube-api-access-cwlbm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.408917 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408921 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408931 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c2a84c-9317-4577-953a-2191b2874cc9-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408941 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408949 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0192173-9c63-4856-802e-48190aae0942-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408957 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0192173-9c63-4856-802e-48190aae0942-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408966 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3f2691-eb8c-413b-b17c-5938212aeec1-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408974 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vv6\" (UniqueName: \"kubernetes.io/projected/83c2a84c-9317-4577-953a-2191b2874cc9-kube-api-access-d6vv6\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408983 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c2a84c-9317-4577-953a-2191b2874cc9-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.408992 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.409162 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.409000 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0192173-9c63-4856-802e-48190aae0942-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:38.532770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.532686 4510 generic.go:356] "Generic (PLEG): container finished" podID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerID="53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87" exitCode=0 Nov 02 21:49:38.532770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.532742 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" Nov 02 21:49:38.532770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.532749 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" event={"ID":"cf3f2691-eb8c-413b-b17c-5938212aeec1","Type":"ContainerDied","Data":"53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87"} Nov 02 21:49:38.533438 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.532834 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9" event={"ID":"cf3f2691-eb8c-413b-b17c-5938212aeec1","Type":"ContainerDied","Data":"64827458ecdc16564a1f9b0eb06acca2fe8fb2576b4ba46d78139890f0d073ed"} Nov 02 21:49:38.533438 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.532851 4510 scope.go:122] "RemoveContainer" containerID="53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87" Nov 02 21:49:38.535977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.535951 4510 generic.go:356] "Generic (PLEG): container finished" podID="a0192173-9c63-4856-802e-48190aae0942" containerID="94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15" exitCode=0 Nov 02 21:49:38.536168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.536147 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" event={"ID":"a0192173-9c63-4856-802e-48190aae0942","Type":"ContainerDied","Data":"94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15"} Nov 02 21:49:38.536218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.536196 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" event={"ID":"a0192173-9c63-4856-802e-48190aae0942","Type":"ContainerDied","Data":"a995d7e99697de2d5df2608781b9d14b976f8206192079c2237e5d085c4682c5"} Nov 02 21:49:38.536286 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.536274 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb" Nov 02 21:49:38.551160 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.549349 4510 generic.go:356] "Generic (PLEG): container finished" podID="83c2a84c-9317-4577-953a-2191b2874cc9" containerID="7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd" exitCode=0 Nov 02 21:49:38.551160 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.550203 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" Nov 02 21:49:38.555816 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.555653 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" event={"ID":"83c2a84c-9317-4577-953a-2191b2874cc9","Type":"ContainerDied","Data":"7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd"} Nov 02 21:49:38.555943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.555889 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr" event={"ID":"83c2a84c-9317-4577-953a-2191b2874cc9","Type":"ContainerDied","Data":"891e6fb0f5a1afc897ab70b9421410e15736b47849c8290962db188e9d51f0df"} Nov 02 21:49:38.565895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.565095 4510 scope.go:122] "RemoveContainer" containerID="53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87" Nov 02 21:49:38.565895 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:38.565408 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87\": container with ID starting with 53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87 not found: ID does not exist" containerID="53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87" Nov 02 21:49:38.565895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.565441 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87"} err="failed to get container status \"53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87\": rpc error: code = NotFound desc = could not find container \"53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87\": container with ID starting with 53c167054a0c8da57812549baa88704b07891e6bd173d18a664fe3515f136a87 not found: ID does not exist" Nov 02 21:49:38.565895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.565463 4510 scope.go:122] "RemoveContainer" containerID="94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15" Nov 02 21:49:38.578848 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.578826 4510 scope.go:122] "RemoveContainer" containerID="94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15" Nov 02 21:49:38.579163 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:38.579141 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15\": container with ID starting with 94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15 not found: ID does not exist" containerID="94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15" Nov 02 21:49:38.579259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.579170 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15"} err="failed to get container status \"94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15\": rpc error: code = NotFound desc = could not find container \"94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15\": container with ID starting with 94010e507b3d1a6ef0d9e62144316c66b4ec5f473a83621eb1bba92761d94d15 not found: ID does not exist" Nov 02 21:49:38.579259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.579186 4510 scope.go:122] "RemoveContainer" containerID="7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd" Nov 02 21:49:38.589777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.589756 4510 scope.go:122] "RemoveContainer" containerID="7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd" Nov 02 21:49:38.590110 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:38.590075 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd\": container with ID starting with 7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd not found: ID does not exist" containerID="7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd" Nov 02 21:49:38.590202 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.590119 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd"} err="failed to get container status \"7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd\": rpc error: code = NotFound desc = could not find container \"7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd\": container with ID starting with 7272ab7d19db66f5b8ad48c40b14f02856ba78f1132b9df43394761fc0bea7bd not found: ID does not exist" Nov 02 21:49:38.776107 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.776070 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb"] Nov 02 21:49:38.779882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.779853 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-nwvxb"] Nov 02 21:49:38.884673 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.884649 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9"] Nov 02 21:49:38.889218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.889192 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-cflq9"] Nov 02 21:49:38.994928 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.994900 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr"] Nov 02 21:49:38.999370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:38.999344 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-5451a5-openshift-default-9b4d8df47-khkfr"] Nov 02 21:49:40.167728 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.167695 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" path="/var/lib/kubelet/pods/83c2a84c-9317-4577-953a-2191b2874cc9/volumes" Nov 02 21:49:40.168201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.168163 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0192173-9c63-4856-802e-48190aae0942" path="/var/lib/kubelet/pods/a0192173-9c63-4856-802e-48190aae0942/volumes" Nov 02 21:49:40.168618 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.168601 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" path="/var/lib/kubelet/pods/cf3f2691-eb8c-413b-b17c-5938212aeec1/volumes" Nov 02 21:49:40.183416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.183379 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd"] Nov 02 21:49:40.183994 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.183979 4510 cpu_manager.go:414] "containerMap: removing container" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" Nov 02 21:49:40.184058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.183996 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" Nov 02 21:49:40.184058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.184018 4510 cpu_manager.go:414] "containerMap: removing container" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" Nov 02 21:49:40.184058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.184023 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" Nov 02 21:49:40.184058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.184033 4510 cpu_manager.go:414] "containerMap: removing container" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" Nov 02 21:49:40.184058 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.184038 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" Nov 02 21:49:40.184222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.184160 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0192173-9c63-4856-802e-48190aae0942" containerName="istio-proxy" Nov 02 21:49:40.184222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.184171 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3f2691-eb8c-413b-b17c-5938212aeec1" containerName="istio-proxy" Nov 02 21:49:40.184222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.184177 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c2a84c-9317-4577-953a-2191b2874cc9" containerName="istio-proxy" Nov 02 21:49:40.189141 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.189122 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.254992 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.254910 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-686bbf-openshift-default-dockercfg-wg489\"" Nov 02 21:49:40.288657 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.288623 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd"] Nov 02 21:49:40.335534 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.335478 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f2a6a-2feb-43ab-b240-d909953c0cb5-istiod-ca-cert\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.335717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.335550 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-certs\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.335717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.335623 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-envoy\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.335717 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.335675 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-data\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.335824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.335726 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxn4z\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-kube-api-access-fxn4z\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.335860 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.335843 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-token\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.335980 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.335964 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-podinfo\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.336012 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.336000 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-socket\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.336082 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.336066 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-credential-socket\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.437687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437610 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f2a6a-2feb-43ab-b240-d909953c0cb5-istiod-ca-cert\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.437687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437652 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-certs\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.437687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437670 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-envoy\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.437687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437686 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-data\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437712 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxn4z\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-kube-api-access-fxn4z\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437776 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-token\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437884 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-podinfo\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.437913 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-socket\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.438077 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-certs\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.438168 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-data\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438365 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.438343 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-socket\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.438420 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-credential-socket\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.438448 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f2a6a-2feb-43ab-b240-d909953c0cb5-istiod-ca-cert\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.438787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.438765 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-credential-socket\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.440038 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.440019 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-envoy\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.440326 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.440306 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-podinfo\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.491558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.491489 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-token\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.491912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.491889 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxn4z\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-kube-api-access-fxn4z\") pod \"e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.504037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.504008 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:40.938153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.938114 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd"] Nov 02 21:49:40.944440 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:40.944392 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467f2a6a_2feb_43ab_b240_d909953c0cb5.slice/crio-b859f8d303da6af3fca281f3691ed0c7a2edf9ede380f8dfdda26a3af0e17a23 WatchSource:0}: Error finding container b859f8d303da6af3fca281f3691ed0c7a2edf9ede380f8dfdda26a3af0e17a23: Status 404 returned error can't find the container with id b859f8d303da6af3fca281f3691ed0c7a2edf9ede380f8dfdda26a3af0e17a23 Nov 02 21:49:40.947192 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.947155 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:40.947278 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.947228 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:40.947278 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:40.947263 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:41.578056 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:41.578022 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" event={"ID":"467f2a6a-2feb-43ab-b240-d909953c0cb5","Type":"ContainerStarted","Data":"ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506"} Nov 02 21:49:41.578552 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:41.578089 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" event={"ID":"467f2a6a-2feb-43ab-b240-d909953c0cb5","Type":"ContainerStarted","Data":"b859f8d303da6af3fca281f3691ed0c7a2edf9ede380f8dfdda26a3af0e17a23"} Nov 02 21:49:41.778724 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:41.778673 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" podStartSLOduration=1.778660575 podStartE2EDuration="1.778660575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:41.731415629 +0000 UTC m=+2207.707713403" watchObservedRunningTime="2026-11-02 21:49:41.778660575 +0000 UTC m=+2207.754958350" Nov 02 21:49:42.459423 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.459390 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7"] Nov 02 21:49:42.464166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.464146 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.504227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.504205 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:42.510152 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.510129 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:42.568681 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.568626 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.568883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.568713 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shst\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-kube-api-access-8shst\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.568883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.568774 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1712603-e038-4107-84e4-2250486933c9-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.568883 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.568845 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.569073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.568894 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.569073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.568927 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1712603-e038-4107-84e4-2250486933c9-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.569073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.568963 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.569073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.569003 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.569073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.569047 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.610721 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.610687 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:42.611616 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.611595 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:49:42.660551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.660487 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7"] Nov 02 21:49:42.670911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.670874 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.670980 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671038 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shst\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-kube-api-access-8shst\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671206 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671113 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1712603-e038-4107-84e4-2250486933c9-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671207 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671322 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671265 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671322 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671314 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1712603-e038-4107-84e4-2250486933c9-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671355 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671489 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671477 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.671847 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671824 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.672011 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.671905 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.672073 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.672008 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.672179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.672154 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.673447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.673399 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1712603-e038-4107-84e4-2250486933c9-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.675230 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.674797 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.675692 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.675669 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1712603-e038-4107-84e4-2250486933c9-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.677151 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.677132 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg"] Nov 02 21:49:42.682781 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.682762 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:42.724253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.724226 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.724636 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.724618 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shst\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-kube-api-access-8shst\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.731792 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.731764 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d0946d-dockercfg-h8bpm\"" Nov 02 21:49:42.732030 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.731789 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d0946d\"" Nov 02 21:49:42.735113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.735087 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg"] Nov 02 21:49:42.788546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.786745 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:42.815737 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.815698 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp"] Nov 02 21:49:42.855589 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.844231 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:42.875153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.874587 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ad13ea-3381-4aaf-8547-9e16c042420c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:42.875153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.874839 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4nbs\" (UniqueName: \"kubernetes.io/projected/08ad13ea-3381-4aaf-8547-9e16c042420c-kube-api-access-b4nbs\") pod \"payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:42.882416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.882326 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp"] Nov 02 21:49:42.976572 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.976478 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4nbs\" (UniqueName: \"kubernetes.io/projected/08ad13ea-3381-4aaf-8547-9e16c042420c-kube-api-access-b4nbs\") pod \"payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:42.976738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.976581 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ad13ea-3381-4aaf-8547-9e16c042420c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:42.976738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.976615 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcr5k\" (UniqueName: \"kubernetes.io/projected/2b9b86b0-9191-4fb3-87fb-944da476a3c4-kube-api-access-gcr5k\") pod \"payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:42.976869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.976746 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9b86b0-9191-4fb3-87fb-944da476a3c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:42.977395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:42.977375 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ad13ea-3381-4aaf-8547-9e16c042420c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:43.029790 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.029763 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4nbs\" (UniqueName: \"kubernetes.io/projected/08ad13ea-3381-4aaf-8547-9e16c042420c-kube-api-access-b4nbs\") pod \"payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:43.077869 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.077828 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcr5k\" (UniqueName: \"kubernetes.io/projected/2b9b86b0-9191-4fb3-87fb-944da476a3c4-kube-api-access-gcr5k\") pod \"payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:43.078015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.077882 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9b86b0-9191-4fb3-87fb-944da476a3c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:43.078505 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.078488 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9b86b0-9191-4fb3-87fb-944da476a3c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:43.135096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.131281 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcr5k\" (UniqueName: \"kubernetes.io/projected/2b9b86b0-9191-4fb3-87fb-944da476a3c4-kube-api-access-gcr5k\") pod \"payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:43.188099 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.188072 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:43.293566 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.293513 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7"] Nov 02 21:49:43.300137 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:43.300102 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1712603_e038_4107_84e4_2250486933c9.slice/crio-6990a1bb662f4aa290e3a7a949173f7916270ac77a8996fa93a272b2a795a517 WatchSource:0}: Error finding container 6990a1bb662f4aa290e3a7a949173f7916270ac77a8996fa93a272b2a795a517: Status 404 returned error can't find the container with id 6990a1bb662f4aa290e3a7a949173f7916270ac77a8996fa93a272b2a795a517 Nov 02 21:49:43.303429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.303391 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:43.303503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.303464 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:43.303503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.303494 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:43.327640 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.327618 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:43.555795 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.555722 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 21:49:43.565641 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.565608 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp"] Nov 02 21:49:43.572279 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:43.572250 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9b86b0_9191_4fb3_87fb_944da476a3c4.slice/crio-c20c73fd94098e66d04a57f48e72b28dddffbaffaef43d2556a8fd5f917f5c6c WatchSource:0}: Error finding container c20c73fd94098e66d04a57f48e72b28dddffbaffaef43d2556a8fd5f917f5c6c: Status 404 returned error can't find the container with id c20c73fd94098e66d04a57f48e72b28dddffbaffaef43d2556a8fd5f917f5c6c Nov 02 21:49:43.620223 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.620194 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" event={"ID":"e1712603-e038-4107-84e4-2250486933c9","Type":"ContainerStarted","Data":"ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b"} Nov 02 21:49:43.620628 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.620262 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" event={"ID":"e1712603-e038-4107-84e4-2250486933c9","Type":"ContainerStarted","Data":"6990a1bb662f4aa290e3a7a949173f7916270ac77a8996fa93a272b2a795a517"} Nov 02 21:49:43.621588 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.621562 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" event={"ID":"2b9b86b0-9191-4fb3-87fb-944da476a3c4","Type":"ContainerStarted","Data":"c20c73fd94098e66d04a57f48e72b28dddffbaffaef43d2556a8fd5f917f5c6c"} Nov 02 21:49:43.708159 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.708113 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg"] Nov 02 21:49:43.714713 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:43.714679 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ad13ea_3381_4aaf_8547_9e16c042420c.slice/crio-fbd0047603e78eff2df9ea2b946f278af3379ff2faf20e4994943fde1e77522e WatchSource:0}: Error finding container fbd0047603e78eff2df9ea2b946f278af3379ff2faf20e4994943fde1e77522e: Status 404 returned error can't find the container with id fbd0047603e78eff2df9ea2b946f278af3379ff2faf20e4994943fde1e77522e Nov 02 21:49:43.780350 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.780299 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podStartSLOduration=1.7802836050000002 podStartE2EDuration="1.780283605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:43.776672708 +0000 UTC m=+2209.752970513" watchObservedRunningTime="2026-11-02 21:49:43.780283605 +0000 UTC m=+2209.756581380" Nov 02 21:49:43.788246 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:43.788223 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:44.651419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.651379 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" event={"ID":"2b9b86b0-9191-4fb3-87fb-944da476a3c4","Type":"ContainerStarted","Data":"0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c"} Nov 02 21:49:44.651927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.651729 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:44.653595 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.653567 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" event={"ID":"08ad13ea-3381-4aaf-8547-9e16c042420c","Type":"ContainerStarted","Data":"546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99"} Nov 02 21:49:44.653734 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.653610 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" event={"ID":"08ad13ea-3381-4aaf-8547-9e16c042420c","Type":"ContainerStarted","Data":"fbd0047603e78eff2df9ea2b946f278af3379ff2faf20e4994943fde1e77522e"} Nov 02 21:49:44.653844 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.653825 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:44.762496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.762445 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" podStartSLOduration=2.762429332 podStartE2EDuration="2.762429332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:44.761469664 +0000 UTC m=+2210.737767439" watchObservedRunningTime="2026-11-02 21:49:44.762429332 +0000 UTC m=+2210.738727106" Nov 02 21:49:44.789270 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.789234 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 21:49:44.789454 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.789299 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 21:49:44.867589 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:44.867513 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" podStartSLOduration=2.867499515 podStartE2EDuration="2.867499515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:44.866962682 +0000 UTC m=+2210.843260458" watchObservedRunningTime="2026-11-02 21:49:44.867499515 +0000 UTC m=+2210.843797290" Nov 02 21:49:45.559748 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:49:45.559708 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:49:45.789104 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:45.789068 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 21:49:45.789581 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:45.789126 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 21:49:46.787956 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:46.787923 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 21:49:46.788146 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:46.787976 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 21:49:47.787940 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:47.787908 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 21:49:47.788290 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:47.787966 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 21:49:48.787972 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:48.787937 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 21:49:48.788418 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:48.787995 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 21:49:49.792546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:49.792464 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 21:49:49.793066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:49.792602 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 21:49:50.216455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.216169 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm"] Nov 02 21:49:50.221837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.221806 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.268566 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.268147 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm"] Nov 02 21:49:50.306512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.306463 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-686bbf\"" Nov 02 21:49:50.306512 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.306502 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-686bbf-dockercfg-7gcvx\"" Nov 02 21:49:50.315217 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.315175 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a5bef8-78a8-4e8e-9112-282d394069b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.315419 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.315399 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbph\" (UniqueName: \"kubernetes.io/projected/76a5bef8-78a8-4e8e-9112-282d394069b4-kube-api-access-5rbph\") pod \"payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.374709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.374683 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv"] Nov 02 21:49:50.389381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.389351 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.416798 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.416768 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbph\" (UniqueName: \"kubernetes.io/projected/76a5bef8-78a8-4e8e-9112-282d394069b4-kube-api-access-5rbph\") pod \"payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.416949 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.416932 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a5bef8-78a8-4e8e-9112-282d394069b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.417620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.417603 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a5bef8-78a8-4e8e-9112-282d394069b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.431627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.431602 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv"] Nov 02 21:49:50.469647 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.469572 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbph\" (UniqueName: \"kubernetes.io/projected/76a5bef8-78a8-4e8e-9112-282d394069b4-kube-api-access-5rbph\") pod \"payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.518423 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.518398 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.518620 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.518602 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4gl\" (UniqueName: \"kubernetes.io/projected/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-kube-api-access-sl4gl\") pod \"payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.545852 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.545828 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:50.623033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.620707 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.623033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.620937 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4gl\" (UniqueName: \"kubernetes.io/projected/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-kube-api-access-sl4gl\") pod \"payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.623033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.621641 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.673671 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.673639 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4gl\" (UniqueName: \"kubernetes.io/projected/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-kube-api-access-sl4gl\") pod \"payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.703845 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.703817 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:50.793234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.793206 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:50.793652 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.793633 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:50.795292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.795271 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:49:50.919365 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:50.919333 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm"] Nov 02 21:49:50.926185 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:50.926162 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a5bef8_78a8_4e8e_9112_282d394069b4.slice/crio-b687f284740336a9827fd8c07d9a55ea38a24952286f6aff8d089edee9564f38 WatchSource:0}: Error finding container b687f284740336a9827fd8c07d9a55ea38a24952286f6aff8d089edee9564f38: Status 404 returned error can't find the container with id b687f284740336a9827fd8c07d9a55ea38a24952286f6aff8d089edee9564f38 Nov 02 21:49:51.080824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.080743 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv"] Nov 02 21:49:51.088733 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:51.088703 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59f0cae_4ae9_4841_8bca_b9d38ca799bc.slice/crio-eaaaadce2099ad196451b773f5adc6455669bc228439eb221803b215cee1fe1e WatchSource:0}: Error finding container eaaaadce2099ad196451b773f5adc6455669bc228439eb221803b215cee1fe1e: Status 404 returned error can't find the container with id eaaaadce2099ad196451b773f5adc6455669bc228439eb221803b215cee1fe1e Nov 02 21:49:51.735317 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.735257 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" event={"ID":"76a5bef8-78a8-4e8e-9112-282d394069b4","Type":"ContainerStarted","Data":"4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a"} Nov 02 21:49:51.735547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.735351 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" event={"ID":"76a5bef8-78a8-4e8e-9112-282d394069b4","Type":"ContainerStarted","Data":"b687f284740336a9827fd8c07d9a55ea38a24952286f6aff8d089edee9564f38"} Nov 02 21:49:51.736808 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.736773 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:49:51.741619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.740790 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" event={"ID":"c59f0cae-4ae9-4841-8bca-b9d38ca799bc","Type":"ContainerStarted","Data":"179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440"} Nov 02 21:49:51.741619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.740848 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:49:51.741619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.740872 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" event={"ID":"c59f0cae-4ae9-4841-8bca-b9d38ca799bc","Type":"ContainerStarted","Data":"eaaaadce2099ad196451b773f5adc6455669bc228439eb221803b215cee1fe1e"} Nov 02 21:49:51.845310 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.845257 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" podStartSLOduration=1.84524001 podStartE2EDuration="1.84524001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:51.841017546 +0000 UTC m=+2217.817315346" watchObservedRunningTime="2026-11-02 21:49:51.84524001 +0000 UTC m=+2217.821537786" Nov 02 21:49:51.951222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:51.951136 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" podStartSLOduration=1.9511184940000001 podStartE2EDuration="1.951118494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:51.945769527 +0000 UTC m=+2217.922067299" watchObservedRunningTime="2026-11-02 21:49:51.951118494 +0000 UTC m=+2217.927416273" Nov 02 21:49:55.664498 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:55.664468 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:49:55.664920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:55.664552 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:49:56.603100 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.603074 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj"] Nov 02 21:49:56.603312 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.603282 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" podUID="140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" containerName="maas-api" containerID="cri-o://599b92039aaec497f1afff10d1f95b09bcbe10693fea7b870063bc583736b611" gracePeriod=30 Nov 02 21:49:56.775735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.775704 4510 generic.go:356] "Generic (PLEG): container finished" podID="140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" containerID="599b92039aaec497f1afff10d1f95b09bcbe10693fea7b870063bc583736b611" exitCode=0 Nov 02 21:49:56.775735 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.775739 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" event={"ID":"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8","Type":"ContainerDied","Data":"599b92039aaec497f1afff10d1f95b09bcbe10693fea7b870063bc583736b611"} Nov 02 21:49:56.889293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.889259 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:49:56.933429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.933398 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-maas-api-tls\") pod \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " Nov 02 21:49:56.933560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.933440 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-kube-api-access-2ks27\" (UniqueName: \"kubernetes.io/projected/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-kube-api-access-2ks27\") pod \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " Nov 02 21:49:56.933621 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.933600 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-openshift-service-ca\") pod \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " Nov 02 21:49:56.933659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.933621 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-metrics-certs\") pod \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\" (UID: \"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8\") " Nov 02 21:49:56.933950 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.933926 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-openshift-service-ca" pod "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" (UID: "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:49:56.934141 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.934121 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:56.935691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.935668 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-metrics-certs" pod "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" (UID: "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:56.935949 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.935933 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-maas-api-tls" pod "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" (UID: "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:49:56.935990 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:56.935970 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-kube-api-access-2ks27" pod "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" (UID: "140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8"). InnerVolumeSpecName "kube-api-access-2ks27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:49:57.035241 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.035091 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:57.035241 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.035119 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:57.035241 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.035133 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ks27\" (UniqueName: \"kubernetes.io/projected/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8-kube-api-access-2ks27\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:49:57.119548 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.119506 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7167-d9d6b6"] Nov 02 21:49:57.120338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.120316 4510 cpu_manager.go:414] "containerMap: removing container" podUID="140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" containerName="maas-api" Nov 02 21:49:57.120338 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.120339 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" containerName="maas-api" Nov 02 21:49:57.120538 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.120511 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" containerName="maas-api" Nov 02 21:49:57.124292 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.124273 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:49:57.136675 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.136647 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcf52\" (UniqueName: \"kubernetes.io/projected/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce-kube-api-access-tcf52\") pod \"test-curl-7167-d9d6b6\" (UID: \"6ffa4e76-eab2-4c6c-8372-84c498b0d4ce\") " pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:49:57.172888 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.172848 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9vlh9\"" Nov 02 21:49:57.178086 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.178054 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7167-d9d6b6"] Nov 02 21:49:57.238017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.237985 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcf52\" (UniqueName: \"kubernetes.io/projected/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce-kube-api-access-tcf52\") pod \"test-curl-7167-d9d6b6\" (UID: \"6ffa4e76-eab2-4c6c-8372-84c498b0d4ce\") " pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:49:57.290760 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.290735 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcf52\" (UniqueName: \"kubernetes.io/projected/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce-kube-api-access-tcf52\") pod \"test-curl-7167-d9d6b6\" (UID: \"6ffa4e76-eab2-4c6c-8372-84c498b0d4ce\") " pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:49:57.437406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.437329 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:49:57.560592 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.560564 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg"] Nov 02 21:49:57.566219 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.566199 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.642668 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.642613 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.642837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.642710 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.642837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.642773 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.642968 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.642858 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.642968 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.642911 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhhj\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-kube-api-access-cwhhj\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.642968 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.642949 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.643102 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.643000 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.643102 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.643076 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.643201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.643129 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.661212 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.661181 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg"] Nov 02 21:49:57.744118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744045 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744096 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744144 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744178 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhhj\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-kube-api-access-cwhhj\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744209 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744258 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744340 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744320 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744365 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744453 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744577 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-credential-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744771 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744611 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-certs\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744812 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istiod-ca-cert\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.744826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.744813 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-data\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.745183 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.745150 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-socket\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.747009 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.746988 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-envoy\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.747237 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.747219 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-podinfo\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.784555 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.784531 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" event={"ID":"140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8","Type":"ContainerDied","Data":"afe7f8d410e79ca455acb19ab9a7a9501f80cb731aeb585269299d8b675135a3"} Nov 02 21:49:57.784937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.784584 4510 scope.go:122] "RemoveContainer" containerID="599b92039aaec497f1afff10d1f95b09bcbe10693fea7b870063bc583736b611" Nov 02 21:49:57.784937 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.784587 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj" Nov 02 21:49:57.797132 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.797100 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-token\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.797210 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.797184 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhhj\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-kube-api-access-cwhhj\") pod \"e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.808538 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.808504 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7167-d9d6b6"] Nov 02 21:49:57.815224 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:57.815203 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ffa4e76_eab2_4c6c_8372_84c498b0d4ce.slice/crio-6a867a003a7144c674f2f351a2beb068345f799022d87ecc8a9a319edbd72b79 WatchSource:0}: Error finding container 6a867a003a7144c674f2f351a2beb068345f799022d87ecc8a9a319edbd72b79: Status 404 returned error can't find the container with id 6a867a003a7144c674f2f351a2beb068345f799022d87ecc8a9a319edbd72b79 Nov 02 21:49:57.880285 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.880263 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:57.943037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.942979 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj"] Nov 02 21:49:57.947939 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:57.947913 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-32de96-664d5b767f-9zjdj"] Nov 02 21:49:58.156105 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.156072 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8" path="/var/lib/kubelet/pods/140c5dd2-c6bb-433a-92b8-d0f2d2a57ec8/volumes" Nov 02 21:49:58.255904 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.255873 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg"] Nov 02 21:49:58.264501 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:49:58.264461 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643e5d7e_bab7_4fdc_b2ba_b8791ea410f8.slice/crio-ceb0e0c588d44bd6a07d40d3ed7004ed20a6eb7e589faddaf6343d7489b34726 WatchSource:0}: Error finding container ceb0e0c588d44bd6a07d40d3ed7004ed20a6eb7e589faddaf6343d7489b34726: Status 404 returned error can't find the container with id ceb0e0c588d44bd6a07d40d3ed7004ed20a6eb7e589faddaf6343d7489b34726 Nov 02 21:49:58.267437 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.267404 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:58.267554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.267472 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:58.267554 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.267502 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:49:58.793935 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.793905 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" event={"ID":"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8","Type":"ContainerStarted","Data":"fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9"} Nov 02 21:49:58.794453 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.793962 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" event={"ID":"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8","Type":"ContainerStarted","Data":"ceb0e0c588d44bd6a07d40d3ed7004ed20a6eb7e589faddaf6343d7489b34726"} Nov 02 21:49:58.796762 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.796734 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7167-d9d6b6" event={"ID":"6ffa4e76-eab2-4c6c-8372-84c498b0d4ce","Type":"ContainerStarted","Data":"6a867a003a7144c674f2f351a2beb068345f799022d87ecc8a9a319edbd72b79"} Nov 02 21:49:58.880341 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.880320 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:49:58.904611 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:58.904564 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" podStartSLOduration=1.9045517859999999 podStartE2EDuration="1.904551786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:58.902142054 +0000 UTC m=+2224.878439826" watchObservedRunningTime="2026-11-02 21:49:58.904551786 +0000 UTC m=+2224.880849561" Nov 02 21:49:59.804890 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:59.804858 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7167-d9d6b6" event={"ID":"6ffa4e76-eab2-4c6c-8372-84c498b0d4ce","Type":"ContainerStarted","Data":"80ed010cecd7320f8e7bd45920cb188257e6361a027fa472286235f08ea118a7"} Nov 02 21:49:59.883074 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:59.883041 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:49:59.883264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:59.883094 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:49:59.912255 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:49:59.912205 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7167-d9d6b6" podStartSLOduration=1.595698917 podStartE2EDuration="2.912191572s" totalImagesPullingTime="1.316492655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:49:59.908618633 +0000 UTC m=+2225.884916410" watchObservedRunningTime="2026-11-02 21:49:59.912191572 +0000 UTC m=+2225.888489346" Nov 02 21:50:00.880819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:00.880785 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:50:00.881301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:00.880856 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:50:01.461672 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:01.461628 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7167-d9d6b6"] Nov 02 21:50:01.465428 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:01.465397 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7167-d9d6b6"] Nov 02 21:50:01.819269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:01.819177 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7167-d9d6b6" podUID="6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" containerName="test-curl-7167-d9d6b6" containerID="cri-o://80ed010cecd7320f8e7bd45920cb188257e6361a027fa472286235f08ea118a7" gracePeriod=2 Nov 02 21:50:01.866425 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:01.866387 4510 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7167-d9d6b6\" is forbidden: User \"system:node:ip-10-0-133-33.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-33.us-west-2.compute.internal' and this object" podUID="6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:50:01.881631 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:01.881597 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:50:01.881982 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:01.881645 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:50:02.617251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.617201 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm"] Nov 02 21:50:02.617529 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.617480 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" podUID="8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" containerName="maas-api" containerID="cri-o://ac2a42151c4664785afa1c675ee402a87a6484d84cf8ac4b728e0ef520ad3262" gracePeriod=30 Nov 02 21:50:02.713472 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:02.713438 4510 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bca1aa3_0f2d_4315_8ba7_5cbbf40ae6d1.slice/crio-ac2a42151c4664785afa1c675ee402a87a6484d84cf8ac4b728e0ef520ad3262.scope\": RecentStats: unable to find data in memory cache]" Nov 02 21:50:02.752400 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.751483 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 21:50:02.881160 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.880688 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:50:02.881160 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.880742 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:50:02.888820 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.888785 4510 generic.go:356] "Generic (PLEG): container finished" podID="8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" containerID="ac2a42151c4664785afa1c675ee402a87a6484d84cf8ac4b728e0ef520ad3262" exitCode=0 Nov 02 21:50:02.889486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.889462 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" event={"ID":"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1","Type":"ContainerDied","Data":"ac2a42151c4664785afa1c675ee402a87a6484d84cf8ac4b728e0ef520ad3262"} Nov 02 21:50:02.992148 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:02.992115 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:50:03.017562 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.017532 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-kube-api-access-8xpj2\" (UniqueName: \"kubernetes.io/projected/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-kube-api-access-8xpj2\") pod \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " Nov 02 21:50:03.017725 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.017624 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-openshift-service-ca\") pod \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " Nov 02 21:50:03.017804 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.017790 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls\") pod \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " Nov 02 21:50:03.017866 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.017818 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-metrics-certs\") pod \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\" (UID: \"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1\") " Nov 02 21:50:03.018008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.017983 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-openshift-service-ca" pod "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" (UID: "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:03.018378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.018355 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:03.019703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.019675 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-kube-api-access-8xpj2" pod "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" (UID: "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1"). InnerVolumeSpecName "kube-api-access-8xpj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:03.019818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.019798 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-metrics-certs" pod "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" (UID: "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:03.019903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.019887 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls" pod "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" (UID: "8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:03.148378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.122596 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:03.148378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.122636 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:03.148378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.122652 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xpj2\" (UniqueName: \"kubernetes.io/projected/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1-kube-api-access-8xpj2\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:03.767545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.767011 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 21:50:03.883466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.881925 4510 patch_prober.go:29] interesting pod/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 02 21:50:03.883466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.882005 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 02 21:50:03.901096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.901064 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" event={"ID":"8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1","Type":"ContainerDied","Data":"e209c39ccca2d99d19f88c20fa984944256b56bce8fe18f0dd97ccf04df46311"} Nov 02 21:50:03.901556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.901130 4510 scope.go:122] "RemoveContainer" containerID="ac2a42151c4664785afa1c675ee402a87a6484d84cf8ac4b728e0ef520ad3262" Nov 02 21:50:03.901556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.901073 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm" Nov 02 21:50:03.909874 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:03.909816 4510 generic.go:356] "Generic (PLEG): container finished" podID="6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" containerID="80ed010cecd7320f8e7bd45920cb188257e6361a027fa472286235f08ea118a7" exitCode=137 Nov 02 21:50:04.056988 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.056959 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm"] Nov 02 21:50:04.062194 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.062164 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a1bf7d-548768dc87-7lqqm"] Nov 02 21:50:04.087469 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.087444 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:50:04.157981 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.157948 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" path="/var/lib/kubelet/pods/8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1/volumes" Nov 02 21:50:04.167770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.167747 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce-kube-api-access-tcf52\" (UniqueName: \"kubernetes.io/projected/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce-kube-api-access-tcf52\") pod \"6ffa4e76-eab2-4c6c-8372-84c498b0d4ce\" (UID: \"6ffa4e76-eab2-4c6c-8372-84c498b0d4ce\") " Nov 02 21:50:04.169845 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.169822 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce-kube-api-access-tcf52" pod "6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" (UID: "6ffa4e76-eab2-4c6c-8372-84c498b0d4ce"). InnerVolumeSpecName "kube-api-access-tcf52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:04.270107 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.270068 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcf52\" (UniqueName: \"kubernetes.io/projected/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce-kube-api-access-tcf52\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:04.921242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.921205 4510 scope.go:122] "RemoveContainer" containerID="80ed010cecd7320f8e7bd45920cb188257e6361a027fa472286235f08ea118a7" Nov 02 21:50:04.921242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.921234 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7167-d9d6b6" Nov 02 21:50:04.996766 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.996721 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:50:04.997458 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.997436 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:50:04.998902 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:04.998881 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:50:06.155423 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:06.155393 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" path="/var/lib/kubelet/pods/6ffa4e76-eab2-4c6c-8372-84c498b0d4ce/volumes" Nov 02 21:50:15.531324 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:15.531288 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:50:23.334439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:23.334400 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm"] Nov 02 21:50:28.180256 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:28.180224 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:50:28.334919 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:28.334879 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" podUID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" containerName="payload-processing" containerID="cri-o://99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11" gracePeriod=25 Nov 02 21:50:34.457670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.457633 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs"] Nov 02 21:50:34.458353 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.458335 4510 cpu_manager.go:414] "containerMap: removing container" podUID="8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" containerName="maas-api" Nov 02 21:50:34.458353 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.458354 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" containerName="maas-api" Nov 02 21:50:34.458449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.458380 4510 cpu_manager.go:414] "containerMap: removing container" podUID="6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" containerName="test-curl-7167-d9d6b6" Nov 02 21:50:34.458449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.458385 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" containerName="test-curl-7167-d9d6b6" Nov 02 21:50:34.458544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.458494 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffa4e76-eab2-4c6c-8372-84c498b0d4ce" containerName="test-curl-7167-d9d6b6" Nov 02 21:50:34.458544 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.458514 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bca1aa3-0f2d-4315-8ba7-5cbbf40ae6d1" containerName="maas-api" Nov 02 21:50:34.462809 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.462789 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.558059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.558020 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d0946d-serving-cert\"" Nov 02 21:50:34.561741 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.561711 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs"] Nov 02 21:50:34.612958 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.612924 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5f9f36-7d12-4dab-b335-3cbb45768e62-openshift-service-ca\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.613138 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.613092 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.613211 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.613144 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9r8m\" (UniqueName: \"kubernetes.io/projected/4f5f9f36-7d12-4dab-b335-3cbb45768e62-kube-api-access-k9r8m\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.613305 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.613279 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-metrics-certs\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.715031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.714929 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.715031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.714989 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9r8m\" (UniqueName: \"kubernetes.io/projected/4f5f9f36-7d12-4dab-b335-3cbb45768e62-kube-api-access-k9r8m\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.715270 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.715088 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-metrics-certs\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.715270 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:34.715087 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-serving-cert: secret "maas-api-e2e-api-a-d0946d-serving-cert" not found Nov 02 21:50:34.715376 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.715278 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5f9f36-7d12-4dab-b335-3cbb45768e62-openshift-service-ca\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.715376 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:34.715298 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls podName:4f5f9f36-7d12-4dab-b335-3cbb45768e62 nodeName:}" failed. No retries permitted until 2026-11-02 21:50:35.215276427 +0000 UTC m=+2261.191574182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls") pod "maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" (UID: "4f5f9f36-7d12-4dab-b335-3cbb45768e62") : secret "maas-api-e2e-api-a-d0946d-serving-cert" not found Nov 02 21:50:34.715896 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.715873 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5f9f36-7d12-4dab-b335-3cbb45768e62-openshift-service-ca\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.717722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.717696 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-metrics-certs\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:34.768709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:34.768663 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9r8m\" (UniqueName: \"kubernetes.io/projected/4f5f9f36-7d12-4dab-b335-3cbb45768e62-kube-api-access-k9r8m\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:35.226004 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:35.225962 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:35.228413 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:35.228389 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls\") pod \"maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:35.376153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:35.376120 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:35.753324 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:35.753288 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs"] Nov 02 21:50:35.759796 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:50:35.759750 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5f9f36_7d12_4dab_b335_3cbb45768e62.slice/crio-3bd33348221ceab30366860a678ffbe319aefb4da58b031bec0dc2fbf9b7cff8 WatchSource:0}: Error finding container 3bd33348221ceab30366860a678ffbe319aefb4da58b031bec0dc2fbf9b7cff8: Status 404 returned error can't find the container with id 3bd33348221ceab30366860a678ffbe319aefb4da58b031bec0dc2fbf9b7cff8 Nov 02 21:50:36.193480 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:36.193445 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" event={"ID":"4f5f9f36-7d12-4dab-b335-3cbb45768e62","Type":"ContainerStarted","Data":"3bd33348221ceab30366860a678ffbe319aefb4da58b031bec0dc2fbf9b7cff8"} Nov 02 21:50:37.201192 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:37.201158 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" event={"ID":"4f5f9f36-7d12-4dab-b335-3cbb45768e62","Type":"ContainerStarted","Data":"6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f"} Nov 02 21:50:37.201649 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:37.201217 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:37.308193 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:37.308141 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" podStartSLOduration=2.395288111 podStartE2EDuration="3.308127091s" totalImagesPullingTime="912.83898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:37.306445944 +0000 UTC m=+2263.282743714" watchObservedRunningTime="2026-11-02 21:50:37.308127091 +0000 UTC m=+2263.284424867" Nov 02 21:50:37.422978 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:37.422935 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" podUID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:50:43.213311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:43.213277 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:50:45.532486 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:45.532441 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:50:47.422884 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:47.422850 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" podUID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:50:48.609995 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:48.609936 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:50:48.707558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:48.707505 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-kube-api-access-8vp42\" (UniqueName: \"kubernetes.io/projected/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-kube-api-access-8vp42\") pod \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " Nov 02 21:50:48.707707 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:48.707673 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-plugins-config-volume\") pod \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\" (UID: \"cfaa81dc-d407-4a1e-b9ff-950e1396ab81\") " Nov 02 21:50:48.708015 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:48.707993 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-plugins-config-volume" pod "cfaa81dc-d407-4a1e-b9ff-950e1396ab81" (UID: "cfaa81dc-d407-4a1e-b9ff-950e1396ab81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:48.708267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:48.708252 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:48.709601 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:48.709578 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-kube-api-access-8vp42" pod "cfaa81dc-d407-4a1e-b9ff-950e1396ab81" (UID: "cfaa81dc-d407-4a1e-b9ff-950e1396ab81"). InnerVolumeSpecName "kube-api-access-8vp42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:48.809686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:48.809654 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vp42\" (UniqueName: \"kubernetes.io/projected/cfaa81dc-d407-4a1e-b9ff-950e1396ab81-kube-api-access-8vp42\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:49.278853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.278817 4510 generic.go:356] "Generic (PLEG): container finished" podID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" containerID="99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11" exitCode=0 Nov 02 21:50:49.279041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.278862 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" Nov 02 21:50:49.279041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.278902 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" event={"ID":"cfaa81dc-d407-4a1e-b9ff-950e1396ab81","Type":"ContainerDied","Data":"99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11"} Nov 02 21:50:49.279041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.278954 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm" event={"ID":"cfaa81dc-d407-4a1e-b9ff-950e1396ab81","Type":"ContainerDied","Data":"9d6034f8739ffa28ab41dbdbfa9cabc07a3cd113637e1c79f9dbad39eaecc04f"} Nov 02 21:50:49.279041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.278976 4510 scope.go:122] "RemoveContainer" containerID="99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11" Nov 02 21:50:49.289390 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.289372 4510 scope.go:122] "RemoveContainer" containerID="99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11" Nov 02 21:50:49.289676 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:49.289655 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11\": container with ID starting with 99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11 not found: ID does not exist" containerID="99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11" Nov 02 21:50:49.289736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.289684 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11"} err="failed to get container status \"99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11\": rpc error: code = NotFound desc = could not find container \"99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11\": container with ID starting with 99752b3ea24427bbf659a7de55c1b5f6136553b213d895f5b33286e703762e11 not found: ID does not exist" Nov 02 21:50:49.432750 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.432713 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm"] Nov 02 21:50:49.478551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.478506 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dd6574-79864967d7-f8wcm"] Nov 02 21:50:49.670096 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.670061 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892"] Nov 02 21:50:49.670868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.670851 4510 cpu_manager.go:414] "containerMap: removing container" podUID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" containerName="payload-processing" Nov 02 21:50:49.670868 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.670869 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" containerName="payload-processing" Nov 02 21:50:49.671001 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.670992 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" containerName="payload-processing" Nov 02 21:50:49.674614 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.674596 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.722185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.722156 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.722322 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.722263 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbc8325-78a6-4d47-ad11-b255b5428937-openshift-service-ca\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.722392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.722378 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzbp\" (UniqueName: \"kubernetes.io/projected/abbc8325-78a6-4d47-ad11-b255b5428937-kube-api-access-dnzbp\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.722440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.722430 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-metrics-certs\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.723145 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.723127 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-686bbf-serving-cert\"" Nov 02 21:50:49.728269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.728245 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892"] Nov 02 21:50:49.825190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.825150 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-metrics-certs\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.825334 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.825284 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.825411 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.825358 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbc8325-78a6-4d47-ad11-b255b5428937-openshift-service-ca\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.825469 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:49.825444 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-serving-cert: secret "maas-api-e2e-inf-a-686bbf-serving-cert" not found Nov 02 21:50:49.825565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.825498 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzbp\" (UniqueName: \"kubernetes.io/projected/abbc8325-78a6-4d47-ad11-b255b5428937-kube-api-access-dnzbp\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.825565 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:49.825540 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls podName:abbc8325-78a6-4d47-ad11-b255b5428937 nodeName:}" failed. No retries permitted until 2026-11-02 21:50:50.325505088 +0000 UTC m=+2276.301802856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls") pod "maas-api-e2e-inf-a-686bbf-6589b9d846-29892" (UID: "abbc8325-78a6-4d47-ad11-b255b5428937") : secret "maas-api-e2e-inf-a-686bbf-serving-cert" not found Nov 02 21:50:49.826507 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.826480 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbc8325-78a6-4d47-ad11-b255b5428937-openshift-service-ca\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.830313 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.830291 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-metrics-certs\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:49.879185 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:49.879152 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzbp\" (UniqueName: \"kubernetes.io/projected/abbc8325-78a6-4d47-ad11-b255b5428937-kube-api-access-dnzbp\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:50.167102 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:50.167065 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfaa81dc-d407-4a1e-b9ff-950e1396ab81" path="/var/lib/kubelet/pods/cfaa81dc-d407-4a1e-b9ff-950e1396ab81/volumes" Nov 02 21:50:50.332244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:50.332214 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:50.334674 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:50.334649 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls\") pod \"maas-api-e2e-inf-a-686bbf-6589b9d846-29892\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:50.590763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:50.590669 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:50.968376 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:50.968339 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892"] Nov 02 21:50:50.974622 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:50:50.974590 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbc8325_78a6_4d47_ad11_b255b5428937.slice/crio-0ba225c3ce27415eb7c0c52806b184cc2c87bc5acf99ff4797a2f36b211049bd WatchSource:0}: Error finding container 0ba225c3ce27415eb7c0c52806b184cc2c87bc5acf99ff4797a2f36b211049bd: Status 404 returned error can't find the container with id 0ba225c3ce27415eb7c0c52806b184cc2c87bc5acf99ff4797a2f36b211049bd Nov 02 21:50:51.295760 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:51.295679 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" event={"ID":"abbc8325-78a6-4d47-ad11-b255b5428937","Type":"ContainerStarted","Data":"0ba225c3ce27415eb7c0c52806b184cc2c87bc5acf99ff4797a2f36b211049bd"} Nov 02 21:50:52.304227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:52.304131 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" event={"ID":"abbc8325-78a6-4d47-ad11-b255b5428937","Type":"ContainerStarted","Data":"2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4"} Nov 02 21:50:52.304227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:52.304217 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:50:52.411451 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:52.411396 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" podStartSLOduration=2.425290686 podStartE2EDuration="3.411384704s" totalImagesPullingTime="986.094018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:50:52.410983965 +0000 UTC m=+2278.387281754" watchObservedRunningTime="2026-11-02 21:50:52.411384704 +0000 UTC m=+2278.387682479" Nov 02 21:50:54.247231 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.247200 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t"] Nov 02 21:50:54.247676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.247417 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" podUID="061f9fe1-ea10-43ca-a32a-8d1aa052efc0" containerName="maas-api" containerID="cri-o://23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51" gracePeriod=30 Nov 02 21:50:54.521988 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.521926 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:50:54.589571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.589539 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-kube-api-access-b6r9r\" (UniqueName: \"kubernetes.io/projected/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-kube-api-access-b6r9r\") pod \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " Nov 02 21:50:54.589757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.589583 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-metrics-certs\") pod \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " Nov 02 21:50:54.589757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.589716 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-openshift-service-ca\") pod \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " Nov 02 21:50:54.589899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.589860 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls\") pod \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\" (UID: \"061f9fe1-ea10-43ca-a32a-8d1aa052efc0\") " Nov 02 21:50:54.590133 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.590095 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-openshift-service-ca" pod "061f9fe1-ea10-43ca-a32a-8d1aa052efc0" (UID: "061f9fe1-ea10-43ca-a32a-8d1aa052efc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:50:54.590543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.590492 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:54.592060 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.592031 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-kube-api-access-b6r9r" pod "061f9fe1-ea10-43ca-a32a-8d1aa052efc0" (UID: "061f9fe1-ea10-43ca-a32a-8d1aa052efc0"). InnerVolumeSpecName "kube-api-access-b6r9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:50:54.592277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.592254 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-metrics-certs" pod "061f9fe1-ea10-43ca-a32a-8d1aa052efc0" (UID: "061f9fe1-ea10-43ca-a32a-8d1aa052efc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:54.592389 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.592375 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls" pod "061f9fe1-ea10-43ca-a32a-8d1aa052efc0" (UID: "061f9fe1-ea10-43ca-a32a-8d1aa052efc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:50:54.691763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.691724 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6r9r\" (UniqueName: \"kubernetes.io/projected/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-kube-api-access-b6r9r\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:54.691763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.691753 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:54.691763 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:54.691764 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061f9fe1-ea10-43ca-a32a-8d1aa052efc0-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:50:55.332404 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.332371 4510 generic.go:356] "Generic (PLEG): container finished" podID="061f9fe1-ea10-43ca-a32a-8d1aa052efc0" containerID="23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51" exitCode=0 Nov 02 21:50:55.332953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.332418 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" Nov 02 21:50:55.332953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.332455 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" event={"ID":"061f9fe1-ea10-43ca-a32a-8d1aa052efc0","Type":"ContainerDied","Data":"23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51"} Nov 02 21:50:55.332953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.332513 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t" event={"ID":"061f9fe1-ea10-43ca-a32a-8d1aa052efc0","Type":"ContainerDied","Data":"347f282792784a13eb14fe1b2377b153810b3f526d508806e9ccf0171c2018b3"} Nov 02 21:50:55.332953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.332557 4510 scope.go:122] "RemoveContainer" containerID="23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51" Nov 02 21:50:55.344739 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.344719 4510 scope.go:122] "RemoveContainer" containerID="23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51" Nov 02 21:50:55.345022 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:50:55.344995 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51\": container with ID starting with 23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51 not found: ID does not exist" containerID="23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51" Nov 02 21:50:55.345090 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.345033 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51"} err="failed to get container status \"23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51\": rpc error: code = NotFound desc = could not find container \"23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51\": container with ID starting with 23ac11f3c0eb70909d7689acbb8bf6687b301a03ee59915384321b95f3575c51 not found: ID does not exist" Nov 02 21:50:55.485248 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.485212 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t"] Nov 02 21:50:55.531875 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:55.531847 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6b983e02-545d64b797-9pl7t"] Nov 02 21:50:56.155661 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:56.155628 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061f9fe1-ea10-43ca-a32a-8d1aa052efc0" path="/var/lib/kubelet/pods/061f9fe1-ea10-43ca-a32a-8d1aa052efc0/volumes" Nov 02 21:50:56.613450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:56.613426 4510 scope.go:122] "RemoveContainer" containerID="2a8680e4debaec44f22ade5c483b4f34e72ed705bb38642abd55dca28591ec29" Nov 02 21:50:58.316533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:50:58.316502 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 21:51:15.251065 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.250977 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26"] Nov 02 21:51:15.251947 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.251925 4510 cpu_manager.go:414] "containerMap: removing container" podUID="061f9fe1-ea10-43ca-a32a-8d1aa052efc0" containerName="maas-api" Nov 02 21:51:15.252048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.251948 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061f9fe1-ea10-43ca-a32a-8d1aa052efc0" containerName="maas-api" Nov 02 21:51:15.252144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.252130 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="061f9fe1-ea10-43ca-a32a-8d1aa052efc0" containerName="maas-api" Nov 02 21:51:15.257966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.257945 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.296738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.296712 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26"] Nov 02 21:51:15.348543 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.344683 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c1243c-serving-cert\"" Nov 02 21:51:15.421912 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.421880 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.422098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.421931 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.422166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.422109 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wffmq\" (UniqueName: \"kubernetes.io/projected/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-kube-api-access-wffmq\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.422166 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.422155 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-metrics-certs\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.523911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.523814 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.531818 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.531780 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wffmq\" (UniqueName: \"kubernetes.io/projected/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-kube-api-access-wffmq\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.532097 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.532076 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-metrics-certs\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.532367 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.532349 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.533383 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:15.533363 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-serving-cert: secret "maas-api-e2e-inf-a-c1243c-serving-cert" not found Nov 02 21:51:15.543651 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:15.543629 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls podName:8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e nodeName:}" failed. No retries permitted until 2026-11-02 21:51:16.043605829 +0000 UTC m=+2302.019903589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls") pod "maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" (UID: "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e") : secret "maas-api-e2e-inf-a-c1243c-serving-cert" not found Nov 02 21:51:15.544660 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.544635 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.547387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.547106 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-metrics-certs\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:15.554583 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:15.554546 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:51:15.587117 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:15.587083 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wffmq\" (UniqueName: \"kubernetes.io/projected/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-kube-api-access-wffmq\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:16.141135 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:16.141100 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:16.143798 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:16.143773 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:16.172112 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:16.172086 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:16.556021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:16.555936 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26"] Nov 02 21:51:16.562185 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:51:16.562154 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c94fd2c_6d5e_4366_82ee_9d46ab7f3b6e.slice/crio-de1b3b35ff84729b98e9f1303fcbdc1b0315a9df95b8dad3226d7080387f8127 WatchSource:0}: Error finding container de1b3b35ff84729b98e9f1303fcbdc1b0315a9df95b8dad3226d7080387f8127: Status 404 returned error can't find the container with id de1b3b35ff84729b98e9f1303fcbdc1b0315a9df95b8dad3226d7080387f8127 Nov 02 21:51:17.507464 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:17.507430 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" event={"ID":"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e","Type":"ContainerStarted","Data":"de1b3b35ff84729b98e9f1303fcbdc1b0315a9df95b8dad3226d7080387f8127"} Nov 02 21:51:18.517577 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:18.517531 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" event={"ID":"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e","Type":"ContainerStarted","Data":"d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81"} Nov 02 21:51:18.517989 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:18.517607 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:18.626684 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:18.626628 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" podStartSLOduration=2.773065875 podStartE2EDuration="3.626613712s" totalImagesPullingTime="853.547837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:18.623078819 +0000 UTC m=+2304.599376594" watchObservedRunningTime="2026-11-02 21:51:18.626613712 +0000 UTC m=+2304.602911515" Nov 02 21:51:19.526182 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:19.526112 4510 generic.go:356] "Generic (PLEG): container finished" podID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerID="ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506" exitCode=0 Nov 02 21:51:19.526688 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:19.526195 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" event={"ID":"467f2a6a-2feb-43ab-b240-d909953c0cb5","Type":"ContainerDied","Data":"ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506"} Nov 02 21:51:19.526775 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:19.526708 4510 scope.go:122] "RemoveContainer" containerID="ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506" Nov 02 21:51:19.528213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:19.528183 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:19.528308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:19.528254 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:19.528308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:19.528287 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 21:51:20.554948 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:20.554914 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" event={"ID":"467f2a6a-2feb-43ab-b240-d909953c0cb5","Type":"ContainerStarted","Data":"b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a"} Nov 02 21:51:21.504515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:21.504481 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:51:21.504723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:21.504613 4510 patch_prober.go:29] interesting pod/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:51:21.504723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:21.504701 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:51:22.505235 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:22.505197 4510 patch_prober.go:29] interesting pod/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:51:22.505638 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:22.505277 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:51:23.505321 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:23.505287 4510 patch_prober.go:29] interesting pod/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:51:23.505746 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:23.505342 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:51:24.505265 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:24.505227 4510 patch_prober.go:29] interesting pod/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:51:24.505447 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:24.505289 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:51:24.552578 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:24.552545 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 21:51:25.504488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:25.504446 4510 patch_prober.go:29] interesting pod/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 21:51:25.504686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:25.504506 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 21:51:26.509356 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:26.509324 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:51:26.509776 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:26.509755 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:51:26.510395 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:26.510377 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 21:51:41.731331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:41.731299 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7"] Nov 02 21:51:41.731858 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:41.731492 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" containerID="cri-o://ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f" gracePeriod=30 Nov 02 21:51:45.531133 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:45.531100 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:51:46.995120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:46.995096 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:51:47.134057 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134030 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-podinfo\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134215 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-certs\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134263 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-socket\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134311 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-kube-api-access-zvgh2\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-kube-api-access-zvgh2\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134358 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-credential-socket\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134499 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134461 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-token\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-token\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134589 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134560 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-data\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134589 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134569 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-certs" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.134699 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134625 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-socket" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.134699 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134648 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-credential-socket" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.134699 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134657 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-envoy\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134732 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab97c6c4-9002-4974-af0b-5a88740a614b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab97c6c4-9002-4974-af0b-5a88740a614b-istiod-ca-cert\") pod \"ab97c6c4-9002-4974-af0b-5a88740a614b\" (UID: \"ab97c6c4-9002-4974-af0b-5a88740a614b\") " Nov 02 21:51:47.134853 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.134821 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-data" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.135178 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.135139 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab97c6c4-9002-4974-af0b-5a88740a614b-istiod-ca-cert" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:51:47.135430 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.135403 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.135558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.135435 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.135558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.135452 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.135558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.135464 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.135558 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.135477 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab97c6c4-9002-4974-af0b-5a88740a614b-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.136391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.136363 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-podinfo" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:51:47.136588 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.136569 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-kube-api-access-zvgh2" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "kube-api-access-zvgh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:47.136670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.136656 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-envoy" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:51:47.136819 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.136796 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-token" pod "ab97c6c4-9002-4974-af0b-5a88740a614b" (UID: "ab97c6c4-9002-4974-af0b-5a88740a614b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:51:47.236493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.236451 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.236493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.236484 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.236493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.236502 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvgh2\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-kube-api-access-zvgh2\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.236770 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.236532 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab97c6c4-9002-4974-af0b-5a88740a614b-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:51:47.748535 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.748482 4510 generic.go:356] "Generic (PLEG): container finished" podID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerID="ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f" exitCode=0 Nov 02 21:51:47.748698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.748558 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" Nov 02 21:51:47.748698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.748573 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" event={"ID":"ab97c6c4-9002-4974-af0b-5a88740a614b","Type":"ContainerDied","Data":"ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f"} Nov 02 21:51:47.748698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.748629 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7" event={"ID":"ab97c6c4-9002-4974-af0b-5a88740a614b","Type":"ContainerDied","Data":"10d38a84ccef4c3442f9f42ccbc4de356fd4703bc9c0bcc873db333a85581147"} Nov 02 21:51:47.748698 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.748655 4510 scope.go:122] "RemoveContainer" containerID="ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f" Nov 02 21:51:47.760650 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.760629 4510 scope.go:122] "RemoveContainer" containerID="ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f" Nov 02 21:51:47.760946 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:47.760929 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f\": container with ID starting with ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f not found: ID does not exist" containerID="ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f" Nov 02 21:51:47.760993 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.760957 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f"} err="failed to get container status \"ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f\": rpc error: code = NotFound desc = could not find container \"ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f\": container with ID starting with ba32bbcbf5ec12e6c49a196087b5762fbe2e7e1901737668d0afab0a1189877f not found: ID does not exist" Nov 02 21:51:47.948736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.948651 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7"] Nov 02 21:51:47.952153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:47.952123 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dd6574-openshift-default-5d877f7ccc-9bjf7"] Nov 02 21:51:48.161788 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:48.161745 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" path="/var/lib/kubelet/pods/ab97c6c4-9002-4974-af0b-5a88740a614b/volumes" Nov 02 21:51:49.468450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.468416 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7"] Nov 02 21:51:49.469239 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.469221 4510 cpu_manager.go:414] "containerMap: removing container" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" Nov 02 21:51:49.469287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.469242 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" Nov 02 21:51:49.469397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.469388 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab97c6c4-9002-4974-af0b-5a88740a614b" containerName="istio-proxy" Nov 02 21:51:49.474510 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.474490 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.527501 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.527470 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7"] Nov 02 21:51:49.568533 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.568497 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d4f799-serving-cert\"" Nov 02 21:51:49.569882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.569856 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.569975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.569914 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-metrics-certs\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.570048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.570035 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lv6\" (UniqueName: \"kubernetes.io/projected/a5818bc3-4b3b-4012-8587-ac0c506eb11a-kube-api-access-x4lv6\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.570098 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.570086 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5818bc3-4b3b-4012-8587-ac0c506eb11a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.671923 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.671889 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lv6\" (UniqueName: \"kubernetes.io/projected/a5818bc3-4b3b-4012-8587-ac0c506eb11a-kube-api-access-x4lv6\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.672111 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.671949 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5818bc3-4b3b-4012-8587-ac0c506eb11a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.672111 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.672077 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.672111 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.672111 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-metrics-certs\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.672319 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:49.672214 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-serving-cert: secret "maas-api-e2e-inf-b-d4f799-serving-cert" not found Nov 02 21:51:49.672319 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:49.672289 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls podName:a5818bc3-4b3b-4012-8587-ac0c506eb11a nodeName:}" failed. No retries permitted until 2026-11-02 21:51:50.172267939 +0000 UTC m=+2336.148565693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls") pod "maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" (UID: "a5818bc3-4b3b-4012-8587-ac0c506eb11a") : secret "maas-api-e2e-inf-b-d4f799-serving-cert" not found Nov 02 21:51:49.672739 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.672716 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5818bc3-4b3b-4012-8587-ac0c506eb11a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.674492 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.674473 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-metrics-certs\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:49.728197 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:49.728114 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lv6\" (UniqueName: \"kubernetes.io/projected/a5818bc3-4b3b-4012-8587-ac0c506eb11a-kube-api-access-x4lv6\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:50.181043 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:50.181008 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:50.183405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:50.183381 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls\") pod \"maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:50.388111 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:50.388083 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:50.760888 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:50.760811 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7"] Nov 02 21:51:50.767527 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:51:50.767481 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5818bc3_4b3b_4012_8587_ac0c506eb11a.slice/crio-79bfb41166cf8298f29782ee1233d3e70a95263a90a2754870ab22a53e147b7a WatchSource:0}: Error finding container 79bfb41166cf8298f29782ee1233d3e70a95263a90a2754870ab22a53e147b7a: Status 404 returned error can't find the container with id 79bfb41166cf8298f29782ee1233d3e70a95263a90a2754870ab22a53e147b7a Nov 02 21:51:51.796593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:51.796559 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" event={"ID":"a5818bc3-4b3b-4012-8587-ac0c506eb11a","Type":"ContainerStarted","Data":"79bfb41166cf8298f29782ee1233d3e70a95263a90a2754870ab22a53e147b7a"} Nov 02 21:51:52.074137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:52.073982 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:51:52.808041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:52.808005 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" event={"ID":"a5818bc3-4b3b-4012-8587-ac0c506eb11a","Type":"ContainerStarted","Data":"d2354c14400a68860201f3b8a1f8ecc2302266c3552f83befac2b444c0d589c2"} Nov 02 21:51:52.808585 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:52.808214 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:58.821486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:58.821455 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 21:51:58.929409 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:58.929350 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" podStartSLOduration=9.036293526 podStartE2EDuration="9.929338043s" totalImagesPullingTime="893.044517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:51:52.927017128 +0000 UTC m=+2338.903314949" watchObservedRunningTime="2026-11-02 21:51:58.929338043 +0000 UTC m=+2344.905635853" Nov 02 21:51:59.563264 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.563232 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j"] Nov 02 21:51:59.569215 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.569182 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.603200 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.603176 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-metrics-certs\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.603331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.603309 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.603401 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.603351 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmxf\" (UniqueName: \"kubernetes.io/projected/70ad2a87-ef27-4f57-990d-860fd992a0a3-kube-api-access-jmmxf\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.603493 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.603479 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ad2a87-ef27-4f57-990d-860fd992a0a3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.618366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.618112 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3566f9-serving-cert\"" Nov 02 21:51:59.621291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.621256 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j"] Nov 02 21:51:59.704483 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.704447 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-metrics-certs\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.704683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.704609 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.704683 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.704641 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmxf\" (UniqueName: \"kubernetes.io/projected/70ad2a87-ef27-4f57-990d-860fd992a0a3-kube-api-access-jmmxf\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.704817 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.704720 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ad2a87-ef27-4f57-990d-860fd992a0a3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.705006 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:59.704984 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-serving-cert: secret "maas-api-e2e-inf-a-3566f9-serving-cert" not found Nov 02 21:51:59.705193 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:51:59.705177 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls podName:70ad2a87-ef27-4f57-990d-860fd992a0a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:52:00.205155319 +0000 UTC m=+2346.181453090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls") pod "maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" (UID: "70ad2a87-ef27-4f57-990d-860fd992a0a3") : secret "maas-api-e2e-inf-a-3566f9-serving-cert" not found Nov 02 21:51:59.706216 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.706186 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ad2a87-ef27-4f57-990d-860fd992a0a3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.707925 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.707879 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-metrics-certs\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:51:59.758076 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:51:59.758045 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmxf\" (UniqueName: \"kubernetes.io/projected/70ad2a87-ef27-4f57-990d-860fd992a0a3-kube-api-access-jmmxf\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:52:00.211842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:00.211809 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:52:00.214467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:00.214442 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls\") pod \"maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:52:00.482687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:00.482608 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:52:00.858926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:00.858850 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j"] Nov 02 21:52:00.865490 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:52:00.865463 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ad2a87_ef27_4f57_990d_860fd992a0a3.slice/crio-c683c7a85a3455d34622c4bef35d63d95eb336571a005b267fee427dfff2d0a0 WatchSource:0}: Error finding container c683c7a85a3455d34622c4bef35d63d95eb336571a005b267fee427dfff2d0a0: Status 404 returned error can't find the container with id c683c7a85a3455d34622c4bef35d63d95eb336571a005b267fee427dfff2d0a0 Nov 02 21:52:01.879105 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:01.879062 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" event={"ID":"70ad2a87-ef27-4f57-990d-860fd992a0a3","Type":"ContainerStarted","Data":"52c34f1617a8c60ba7bbdb62c76bc91bb0e982dd8fd3ab58e204dbdede795cad"} Nov 02 21:52:01.879595 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:01.879173 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" event={"ID":"70ad2a87-ef27-4f57-990d-860fd992a0a3","Type":"ContainerStarted","Data":"c683c7a85a3455d34622c4bef35d63d95eb336571a005b267fee427dfff2d0a0"} Nov 02 21:52:01.879595 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:01.879206 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:52:02.000109 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:02.000054 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" podStartSLOduration=2.086455249 podStartE2EDuration="3.000039709s" totalImagesPullingTime="913.58446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:52:01.998804504 +0000 UTC m=+2347.975102278" watchObservedRunningTime="2026-11-02 21:52:02.000039709 +0000 UTC m=+2347.976337500" Nov 02 21:52:07.892470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:07.892443 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 21:52:08.385491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:08.385440 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg"] Nov 02 21:52:08.386132 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:08.386079 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" containerID="cri-o://c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0" gracePeriod=30 Nov 02 21:52:08.407986 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:08.407955 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7"] Nov 02 21:52:08.408179 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:08.408160 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" containerID="cri-o://ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b" gracePeriod=30 Nov 02 21:52:08.428602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:08.428568 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg"] Nov 02 21:52:08.428783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:08.428764 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" containerID="cri-o://fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9" gracePeriod=30 Nov 02 21:52:13.823030 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.822689 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:52:13.828041 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.828017 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:52:13.832308 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.832286 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:52:13.917271 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917243 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-certs\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.917463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917289 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-envoy\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.917463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917324 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-socket\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.917463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917373 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-certs\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.917463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917404 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-envoy\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.917463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917432 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-istio-token\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-istio-token\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917464 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-token\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-token\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917494 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-kube-api-access-cwhhj\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-kube-api-access-cwhhj\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917538 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-podinfo\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917569 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-certs" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917579 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-envoy\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917609 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-token\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-token\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917654 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-socket" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.917761 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917675 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-certs" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917814 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-credential-socket\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917850 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1712603-e038-4107-84e4-2250486933c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1712603-e038-4107-84e4-2250486933c9-istiod-ca-cert\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917892 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1712603-e038-4107-84e4-2250486933c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1712603-e038-4107-84e4-2250486933c9-istio-podinfo\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.917935 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-kube-api-access-8shst\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-kube-api-access-8shst\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918035 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-kube-api-access-q9nhd\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-kube-api-access-q9nhd\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918051 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-credential-socket" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918087 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-credential-socket\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.918176 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918159 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-data\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918204 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-socket\") pod \"e1712603-e038-4107-84e4-2250486933c9\" (UID: \"e1712603-e038-4107-84e4-2250486933c9\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918249 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-podinfo\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918286 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-data\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918316 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-credential-socket\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918350 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istiod-ca-cert\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918381 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istiod-ca-cert\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918435 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-data\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918470 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-certs\") pod \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\" (UID: \"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0\") " Nov 02 21:52:13.918606 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918555 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-socket\") pod \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\" (UID: \"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8\") " Nov 02 21:52:13.919048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918772 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-socket" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.919048 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.918985 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-socket" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.919780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.919218 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-data" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.919780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.919394 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-credential-socket" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.919780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.919450 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-credential-socket" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.919780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.919531 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istiod-ca-cert" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:13.919780 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.919568 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istiod-ca-cert" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.919803 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-data" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920016 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-data" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920023 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920044 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920060 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920075 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920090 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920103 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920118 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920131 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920143 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920145 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920158 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920171 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920184 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:13.920796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920191 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1712603-e038-4107-84e4-2250486933c9-istiod-ca-cert" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:13.920796 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920388 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-certs" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.921081 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.920816 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-token" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:13.921933 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.921903 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-kube-api-access-cwhhj" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "kube-api-access-cwhhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:13.922035 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.921999 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-istio-token" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:13.922035 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.922012 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-envoy" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.922163 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.922074 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-envoy" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.922653 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.922626 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-token" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:13.922743 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.922711 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-kube-api-access-8shst" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "kube-api-access-8shst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:13.922808 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.922742 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-podinfo" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:13.922895 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.922870 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-kube-api-access-q9nhd" pod "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" (UID: "84072bc7-dcbf-4c58-93aa-8fa81ebe31e0"). InnerVolumeSpecName "kube-api-access-q9nhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:13.923503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.923486 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-envoy" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:13.923745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.923731 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-podinfo" pod "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" (UID: "643e5d7e-bab7-4fdc-b2ba-b8791ea410f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:13.924201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.924187 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1712603-e038-4107-84e4-2250486933c9-istio-podinfo" pod "e1712603-e038-4107-84e4-2250486933c9" (UID: "e1712603-e038-4107-84e4-2250486933c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:13.959710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.959643 4510 generic.go:356] "Generic (PLEG): container finished" podID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerID="c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0" exitCode=0 Nov 02 21:52:13.959710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.959706 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" Nov 02 21:52:13.959886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.959729 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" event={"ID":"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0","Type":"ContainerDied","Data":"c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0"} Nov 02 21:52:13.959886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.959791 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg" event={"ID":"84072bc7-dcbf-4c58-93aa-8fa81ebe31e0","Type":"ContainerDied","Data":"d2a56d1146150fdfb03560bcd7d9dbc812a58e6e2efe95c90bdef2e58d4bc461"} Nov 02 21:52:13.959886 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.959822 4510 scope.go:122] "RemoveContainer" containerID="c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0" Nov 02 21:52:13.961140 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.961118 4510 generic.go:356] "Generic (PLEG): container finished" podID="e1712603-e038-4107-84e4-2250486933c9" containerID="ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b" exitCode=0 Nov 02 21:52:13.961229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.961159 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" Nov 02 21:52:13.961229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.961211 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" event={"ID":"e1712603-e038-4107-84e4-2250486933c9","Type":"ContainerDied","Data":"ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b"} Nov 02 21:52:13.961331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.961270 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7" event={"ID":"e1712603-e038-4107-84e4-2250486933c9","Type":"ContainerDied","Data":"6990a1bb662f4aa290e3a7a949173f7916270ac77a8996fa93a272b2a795a517"} Nov 02 21:52:13.962808 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.962786 4510 generic.go:356] "Generic (PLEG): container finished" podID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerID="fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9" exitCode=0 Nov 02 21:52:13.962920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.962835 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" Nov 02 21:52:13.962920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.962894 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" event={"ID":"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8","Type":"ContainerDied","Data":"fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9"} Nov 02 21:52:13.963050 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.962954 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg" event={"ID":"643e5d7e-bab7-4fdc-b2ba-b8791ea410f8","Type":"ContainerDied","Data":"ceb0e0c588d44bd6a07d40d3ed7004ed20a6eb7e589faddaf6343d7489b34726"} Nov 02 21:52:13.971103 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.971004 4510 scope.go:122] "RemoveContainer" containerID="c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0" Nov 02 21:52:13.971442 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:13.971418 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0\": container with ID starting with c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0 not found: ID does not exist" containerID="c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0" Nov 02 21:52:13.971556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.971448 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0"} err="failed to get container status \"c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0\": rpc error: code = NotFound desc = could not find container \"c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0\": container with ID starting with c0881db2b8e68c6b3a4d2a466ff05aa1d149c525357eebbcda02fa409fa9bfd0 not found: ID does not exist" Nov 02 21:52:13.971556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.971467 4510 scope.go:122] "RemoveContainer" containerID="ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b" Nov 02 21:52:13.982061 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.982042 4510 scope.go:122] "RemoveContainer" containerID="ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b" Nov 02 21:52:13.982383 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:13.982357 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b\": container with ID starting with ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b not found: ID does not exist" containerID="ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b" Nov 02 21:52:13.982486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.982394 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b"} err="failed to get container status \"ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b\": rpc error: code = NotFound desc = could not find container \"ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b\": container with ID starting with ba1dcc79c691c0852e7a8c3fd36923763b1a2018a5a86a9b274ff5402f25ee2b not found: ID does not exist" Nov 02 21:52:13.982486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:13.982419 4510 scope.go:122] "RemoveContainer" containerID="fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9" Nov 02 21:52:14.021679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021642 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhhj\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-kube-api-access-cwhhj\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021679 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021679 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021695 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021708 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021720 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1712603-e038-4107-84e4-2250486933c9-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021732 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1712603-e038-4107-84e4-2250486933c9-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021744 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8shst\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-kube-api-access-8shst\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021756 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nhd\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-kube-api-access-q9nhd\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021767 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1712603-e038-4107-84e4-2250486933c9-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021779 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021798 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021810 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021821 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021832 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1712603-e038-4107-84e4-2250486933c9-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.021953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.021844 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:14.029619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.029593 4510 scope.go:122] "RemoveContainer" containerID="fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9" Nov 02 21:52:14.030047 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:14.030017 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9\": container with ID starting with fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9 not found: ID does not exist" containerID="fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9" Nov 02 21:52:14.030047 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.030055 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9"} err="failed to get container status \"fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9\": rpc error: code = NotFound desc = could not find container \"fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9\": container with ID starting with fd8fc4c41760933c6268fa643832995fbaa3648068382d6afb58de70854f1fd9 not found: ID does not exist" Nov 02 21:52:14.145726 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.145691 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg"] Nov 02 21:52:14.159157 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.159128 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-d9dcg"] Nov 02 21:52:14.249200 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.249118 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7"] Nov 02 21:52:14.253561 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.253504 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-76pr7"] Nov 02 21:52:14.415271 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.415238 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg"] Nov 02 21:52:14.421074 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:14.421043 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b983e02-openshift-default-f8946f76f-28fpg"] Nov 02 21:52:15.531363 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:15.531325 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:52:16.156228 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:16.156195 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" path="/var/lib/kubelet/pods/643e5d7e-bab7-4fdc-b2ba-b8791ea410f8/volumes" Nov 02 21:52:16.156754 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:16.156739 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" path="/var/lib/kubelet/pods/84072bc7-dcbf-4c58-93aa-8fa81ebe31e0/volumes" Nov 02 21:52:16.157219 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:16.157204 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1712603-e038-4107-84e4-2250486933c9" path="/var/lib/kubelet/pods/e1712603-e038-4107-84e4-2250486933c9/volumes" Nov 02 21:52:24.992768 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:24.992731 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj"] Nov 02 21:52:24.993296 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:24.992950 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" containerID="cri-o://1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75" gracePeriod=30 Nov 02 21:52:25.670706 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:25.670668 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6"] Nov 02 21:52:25.670943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:25.670912 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" containerID="cri-o://c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee" gracePeriod=30 Nov 02 21:52:25.709584 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:25.709556 4510 patch_prober.go:29] interesting pod/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:52:25.709748 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:25.709634 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:52:29.276085 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:29.276040 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:52:30.275213 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.275152 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:52:30.422075 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422011 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-kube-api-access-882bn\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-kube-api-access-882bn\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422118 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-credential-socket\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422215 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eec368-d117-40a2-adf6-ae868a46b083-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec368-d117-40a2-adf6-ae868a46b083-istiod-ca-cert\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422276 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8eec368-d117-40a2-adf6-ae868a46b083-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec368-d117-40a2-adf6-ae868a46b083-istio-podinfo\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422345 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-istio-token\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422388 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-socket\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422430 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-data\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422454 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-certs\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.422617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.422495 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-envoy\") pod \"f8eec368-d117-40a2-adf6-ae868a46b083\" (UID: \"f8eec368-d117-40a2-adf6-ae868a46b083\") " Nov 02 21:52:30.423514 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.423445 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-certs" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.423663 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.423546 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-credential-socket" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.425738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.424443 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eec368-d117-40a2-adf6-ae868a46b083-istiod-ca-cert" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:30.425738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.424502 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-socket" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.425738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.424712 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-data" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.445168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.445120 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-envoy" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:30.445765 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.445738 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-kube-api-access-882bn" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "kube-api-access-882bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:30.445893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.445880 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-istio-token" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:30.449782 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.448705 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8eec368-d117-40a2-adf6-ae868a46b083-istio-podinfo" pod "f8eec368-d117-40a2-adf6-ae868a46b083" (UID: "f8eec368-d117-40a2-adf6-ae868a46b083"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:30.524302 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524265 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-882bn\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-kube-api-access-882bn\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524302 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524297 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524302 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524307 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8eec368-d117-40a2-adf6-ae868a46b083-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524316 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8eec368-d117-40a2-adf6-ae868a46b083-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524326 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8eec368-d117-40a2-adf6-ae868a46b083-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524334 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524342 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524350 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.524551 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.524372 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8eec368-d117-40a2-adf6-ae868a46b083-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:30.923909 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:30.923882 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:52:31.030515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.030488 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-podinfo\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.030515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.030539 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istiod-ca-cert\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.030749 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.030569 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-token\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.030749 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.030635 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-credential-socket\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.030749 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.030689 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-envoy\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.031066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.030752 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-certs\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.031066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.030831 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-data\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.031211 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031068 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-credential-socket" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:31.031211 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031133 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-kube-api-access-jmqzs\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-kube-api-access-jmqzs\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.031324 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031270 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-socket\") pod \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\" (UID: \"0a9f15c8-aea4-4c05-9945-301b760e5fb7\") " Nov 02 21:52:31.031381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031322 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-certs" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:31.031381 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031366 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-data" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:31.031836 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031737 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istiod-ca-cert" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:52:31.031967 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031934 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.032113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031976 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.032113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031991 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.032113 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.031811 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-socket" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:31.034630 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.034488 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-podinfo" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:52:31.035627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.034887 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-token" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:31.035627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.034916 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-kube-api-access-jmqzs" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "kube-api-access-jmqzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:52:31.035973 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.035927 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-envoy" pod "0a9f15c8-aea4-4c05-9945-301b760e5fb7" (UID: "0a9f15c8-aea4-4c05-9945-301b760e5fb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:52:31.094078 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.094039 4510 generic.go:356] "Generic (PLEG): container finished" podID="f8eec368-d117-40a2-adf6-ae868a46b083" containerID="1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75" exitCode=0 Nov 02 21:52:31.094273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.094101 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" Nov 02 21:52:31.094273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.094122 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" event={"ID":"f8eec368-d117-40a2-adf6-ae868a46b083","Type":"ContainerDied","Data":"1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75"} Nov 02 21:52:31.094273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.094188 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj" event={"ID":"f8eec368-d117-40a2-adf6-ae868a46b083","Type":"ContainerDied","Data":"2ab6b98ff576fe1edd7a32cc6fa2356a02e4d50225b77c063f689db2d603593b"} Nov 02 21:52:31.094273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.094219 4510 scope.go:122] "RemoveContainer" containerID="1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75" Nov 02 21:52:31.096477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.096408 4510 generic.go:356] "Generic (PLEG): container finished" podID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerID="c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee" exitCode=0 Nov 02 21:52:31.096477 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.096464 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" Nov 02 21:52:31.096686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.096460 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" event={"ID":"0a9f15c8-aea4-4c05-9945-301b760e5fb7","Type":"ContainerDied","Data":"c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee"} Nov 02 21:52:31.096686 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.096608 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6" event={"ID":"0a9f15c8-aea4-4c05-9945-301b760e5fb7","Type":"ContainerDied","Data":"a6ff963845e3b12e7b78998748bb3d6e5eaeae9196381d0c75b96204775bfda8"} Nov 02 21:52:31.108837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.108818 4510 scope.go:122] "RemoveContainer" containerID="1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75" Nov 02 21:52:31.109217 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:31.109192 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75\": container with ID starting with 1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75 not found: ID does not exist" containerID="1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75" Nov 02 21:52:31.109331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.109227 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75"} err="failed to get container status \"1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75\": rpc error: code = NotFound desc = could not find container \"1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75\": container with ID starting with 1242186316c3591aa5997a9cd408155e0c7ec2be71bf0339ff1fd3cddab23f75 not found: ID does not exist" Nov 02 21:52:31.109331 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.109251 4510 scope.go:122] "RemoveContainer" containerID="c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee" Nov 02 21:52:31.120893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.120747 4510 scope.go:122] "RemoveContainer" containerID="d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7" Nov 02 21:52:31.143693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.142686 4510 scope.go:122] "RemoveContainer" containerID="c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee" Nov 02 21:52:31.143693 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:31.143384 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee\": container with ID starting with c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee not found: ID does not exist" containerID="c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee" Nov 02 21:52:31.143693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.143427 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee"} err="failed to get container status \"c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee\": rpc error: code = NotFound desc = could not find container \"c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee\": container with ID starting with c8799995632c55ad4153a13301e9d29a5f73ea4f3801edee3ebc2b711605d5ee not found: ID does not exist" Nov 02 21:52:31.143693 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.143454 4510 scope.go:122] "RemoveContainer" containerID="d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7" Nov 02 21:52:31.144021 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:31.143838 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7\": container with ID starting with d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7 not found: ID does not exist" containerID="d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7" Nov 02 21:52:31.144021 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.143878 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7"} err="failed to get container status \"d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7\": rpc error: code = NotFound desc = could not find container \"d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7\": container with ID starting with d46645ce2d3dd3fe6d4b44185a298af136aa752d9fd12c1434074ed19b04dec7 not found: ID does not exist" Nov 02 21:52:31.144625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.144496 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmqzs\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-kube-api-access-jmqzs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.144625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.144544 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.144625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.144563 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.144625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.144577 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.144625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.144591 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.144625 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.144605 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9f15c8-aea4-4c05-9945-301b760e5fb7-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:52:31.252320 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.252271 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6"] Nov 02 21:52:31.257649 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.257617 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-284b8e-openshift-default-77c5d9855-64wq6"] Nov 02 21:52:31.361153 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.361051 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj"] Nov 02 21:52:31.366225 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:31.366189 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c2dd3b-openshift-default-55c8f4465f-dxqcj"] Nov 02 21:52:32.206879 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:32.206830 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" path="/var/lib/kubelet/pods/0a9f15c8-aea4-4c05-9945-301b760e5fb7/volumes" Nov 02 21:52:32.210160 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:52:32.210136 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" path="/var/lib/kubelet/pods/f8eec368-d117-40a2-adf6-ae868a46b083/volumes" Nov 02 21:52:45.533908 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:52:45.533826 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:53:06.653427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.653389 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp"] Nov 02 21:53:06.654072 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654057 4510 cpu_manager.go:414] "containerMap: removing container" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" Nov 02 21:53:06.654118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654073 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" Nov 02 21:53:06.654118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654099 4510 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" Nov 02 21:53:06.654118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654104 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" Nov 02 21:53:06.654118 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654115 4510 cpu_manager.go:414] "containerMap: removing container" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" Nov 02 21:53:06.654242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654119 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" Nov 02 21:53:06.654242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654132 4510 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" Nov 02 21:53:06.654242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654136 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" Nov 02 21:53:06.654242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654144 4510 cpu_manager.go:414] "containerMap: removing container" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" Nov 02 21:53:06.654242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654148 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" Nov 02 21:53:06.654242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654162 4510 cpu_manager.go:414] "containerMap: removing container" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" Nov 02 21:53:06.654242 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654166 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" Nov 02 21:53:06.654449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654275 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" Nov 02 21:53:06.654449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654283 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eec368-d117-40a2-adf6-ae868a46b083" containerName="istio-proxy" Nov 02 21:53:06.654449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654293 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f15c8-aea4-4c05-9945-301b760e5fb7" containerName="istio-proxy" Nov 02 21:53:06.654449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654300 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1712603-e038-4107-84e4-2250486933c9" containerName="istio-proxy" Nov 02 21:53:06.654449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654307 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="643e5d7e-bab7-4fdc-b2ba-b8791ea410f8" containerName="istio-proxy" Nov 02 21:53:06.654449 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.654315 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="84072bc7-dcbf-4c58-93aa-8fa81ebe31e0" containerName="istio-proxy" Nov 02 21:53:06.659450 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.659431 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.711885 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.711862 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp"] Nov 02 21:53:06.731009 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.730984 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-metrics-certs\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.731222 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.731196 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.731347 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.731331 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5bt9\" (UniqueName: \"kubernetes.io/projected/3f233ae1-2af8-4a95-8393-8b9a074b3034-kube-api-access-x5bt9\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.731401 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.731387 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f233ae1-2af8-4a95-8393-8b9a074b3034-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.753856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.753833 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3d2b29-serving-cert\"" Nov 02 21:53:06.832386 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.832359 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bt9\" (UniqueName: \"kubernetes.io/projected/3f233ae1-2af8-4a95-8393-8b9a074b3034-kube-api-access-x5bt9\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.832547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.832406 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f233ae1-2af8-4a95-8393-8b9a074b3034-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.832547 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.832439 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-metrics-certs\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.832648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.832627 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.832769 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:06.832750 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-serving-cert: secret "maas-api-e2e-api-b-3d2b29-serving-cert" not found Nov 02 21:53:06.832849 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:06.832837 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls podName:3f233ae1-2af8-4a95-8393-8b9a074b3034 nodeName:}" failed. No retries permitted until 2026-11-02 21:53:07.332817488 +0000 UTC m=+2413.309115261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls") pod "maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" (UID: "3f233ae1-2af8-4a95-8393-8b9a074b3034") : secret "maas-api-e2e-api-b-3d2b29-serving-cert" not found Nov 02 21:53:06.833174 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.833155 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f233ae1-2af8-4a95-8393-8b9a074b3034-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.834988 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.834966 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-metrics-certs\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:06.886131 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:06.886110 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bt9\" (UniqueName: \"kubernetes.io/projected/3f233ae1-2af8-4a95-8393-8b9a074b3034-kube-api-access-x5bt9\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:07.340695 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:07.340653 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:07.343028 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:07.342999 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls\") pod \"maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:07.572676 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:07.572630 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:07.945429 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:07.945354 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp"] Nov 02 21:53:07.953041 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:53:07.953015 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f233ae1_2af8_4a95_8393_8b9a074b3034.slice/crio-4d7d8936518e75991ef204b7d42ea1b83b1ba8287518142581c890ca377c8960 WatchSource:0}: Error finding container 4d7d8936518e75991ef204b7d42ea1b83b1ba8287518142581c890ca377c8960: Status 404 returned error can't find the container with id 4d7d8936518e75991ef204b7d42ea1b83b1ba8287518142581c890ca377c8960 Nov 02 21:53:08.341529 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:08.341438 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" event={"ID":"3f233ae1-2af8-4a95-8393-8b9a074b3034","Type":"ContainerStarted","Data":"4d7d8936518e75991ef204b7d42ea1b83b1ba8287518142581c890ca377c8960"} Nov 02 21:53:09.349651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:09.349554 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" event={"ID":"3f233ae1-2af8-4a95-8393-8b9a074b3034","Type":"ContainerStarted","Data":"06a384921f52a781cbe9a74367e78ebb8131c783d02d9c51ce9a248211264fb4"} Nov 02 21:53:09.349651 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:09.349644 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:09.457899 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:09.457846 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" podStartSLOduration=2.47730564 podStartE2EDuration="3.457830836s" totalImagesPullingTime="980.525196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:09.454694884 +0000 UTC m=+2415.430992660" watchObservedRunningTime="2026-11-02 21:53:09.457830836 +0000 UTC m=+2415.434128626" Nov 02 21:53:15.320777 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.320743 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h"] Nov 02 21:53:15.325681 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.325661 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.361361 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.361335 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:53:15.421495 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.421460 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7cc1b1\"/\"openshift-service-ca.crt\"" Nov 02 21:53:15.421719 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.421550 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7cc1b1\"/\"kube-root-ca.crt\"" Nov 02 21:53:15.421719 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.421713 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7cc1b1\"/\"default-dockercfg-8lnfp\"" Nov 02 21:53:15.421843 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.421813 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7cc1b1\"/\"test-model-7cc1b1-kserve-self-signed-certs\"" Nov 02 21:53:15.425997 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.425972 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h"] Nov 02 21:53:15.438526 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.438502 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-home\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.438638 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.438547 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-dshm\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.438638 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.438608 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-model-cache\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.438712 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.438636 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tmp-dir\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.438712 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.438675 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlz5w\" (UniqueName: \"kubernetes.io/projected/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-kube-api-access-mlz5w\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.438787 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.438771 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tls-certs\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.532141 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:15.532110 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:53:15.539979 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.539944 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-home\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.539983 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-dshm\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.540030 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-model-cache\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.540065 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tmp-dir\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540321 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.540147 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlz5w\" (UniqueName: \"kubernetes.io/projected/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-kube-api-access-mlz5w\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540321 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.540245 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tls-certs\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540434 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.540352 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-home\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540506 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.540442 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tmp-dir\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.540637 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.540508 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-model-cache\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.542168 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.542143 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-dshm\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.542671 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.542651 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tls-certs\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.598311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.598243 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlz5w\" (UniqueName: \"kubernetes.io/projected/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-kube-api-access-mlz5w\") pod \"test-model-7cc1b1-kserve-7c557d8449-kn42h\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:15.639738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:15.639704 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:16.021008 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:16.020957 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h"] Nov 02 21:53:16.027478 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:53:16.027439 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfde1f82a_a3e0_4d9e_8b0e_9b91e62083a6.slice/crio-9917eef8c644056250cd8e259df9872b99acb1c85552785ea00f0b14fe5134e3 WatchSource:0}: Error finding container 9917eef8c644056250cd8e259df9872b99acb1c85552785ea00f0b14fe5134e3: Status 404 returned error can't find the container with id 9917eef8c644056250cd8e259df9872b99acb1c85552785ea00f0b14fe5134e3 Nov 02 21:53:16.407150 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:16.407108 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" event={"ID":"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6","Type":"ContainerStarted","Data":"0366bf2a983e3ef3023989ede52cc353189f63a07cb27a8b01036bd4bcc085f6"} Nov 02 21:53:16.407556 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:16.407160 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" event={"ID":"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6","Type":"ContainerStarted","Data":"9917eef8c644056250cd8e259df9872b99acb1c85552785ea00f0b14fe5134e3"} Nov 02 21:53:16.515259 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:16.515205 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" podStartSLOduration=1.515191976 podStartE2EDuration="1.515191976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:16.512227548 +0000 UTC m=+2422.488525333" watchObservedRunningTime="2026-11-02 21:53:16.515191976 +0000 UTC m=+2422.491489750" Nov 02 21:53:16.639927 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:16.639857 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:25.640135 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:25.640100 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:25.653252 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:25.653225 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:26.480089 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:26.480056 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:53:35.653570 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.653536 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct"] Nov 02 21:53:35.658149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.658129 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.753184 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.753153 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-11537e-serving-cert\"" Nov 02 21:53:35.756612 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.756587 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct"] Nov 02 21:53:35.779273 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.779244 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-metrics-certs\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.779582 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.779284 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4d6\" (UniqueName: \"kubernetes.io/projected/71a6aa31-bb91-41cd-a91b-d83d613e2275-kube-api-access-vg4d6\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.779657 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.779620 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6aa31-bb91-41cd-a91b-d83d613e2275-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.779718 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.779671 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.881644 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.881610 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6aa31-bb91-41cd-a91b-d83d613e2275-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.881830 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.881655 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.881830 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.881738 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-metrics-certs\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.881830 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.881766 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4d6\" (UniqueName: \"kubernetes.io/projected/71a6aa31-bb91-41cd-a91b-d83d613e2275-kube-api-access-vg4d6\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.881973 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:35.881823 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-serving-cert: secret "maas-api-e2e-inf-b-11537e-serving-cert" not found Nov 02 21:53:35.881973 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:35.881905 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls podName:71a6aa31-bb91-41cd-a91b-d83d613e2275 nodeName:}" failed. No retries permitted until 2026-11-02 21:53:36.381882446 +0000 UTC m=+2442.358180209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls") pod "maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" (UID: "71a6aa31-bb91-41cd-a91b-d83d613e2275") : secret "maas-api-e2e-inf-b-11537e-serving-cert" not found Nov 02 21:53:35.882282 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.882256 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6aa31-bb91-41cd-a91b-d83d613e2275-openshift-service-ca\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.884120 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.884102 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-metrics-certs\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:35.936681 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:35.936614 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4d6\" (UniqueName: \"kubernetes.io/projected/71a6aa31-bb91-41cd-a91b-d83d613e2275-kube-api-access-vg4d6\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:36.388851 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:36.388822 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:36.391316 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:36.391289 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls\") pod \"maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:36.573059 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:36.573025 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:36.964327 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:36.964251 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct"] Nov 02 21:53:36.970914 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:53:36.970889 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a6aa31_bb91_41cd_a91b_d83d613e2275.slice/crio-5e4a7ba40ee614d96442e4fd5542c25e491b00ddce0f26c55ce8cf685a5ea0a1 WatchSource:0}: Error finding container 5e4a7ba40ee614d96442e4fd5542c25e491b00ddce0f26c55ce8cf685a5ea0a1: Status 404 returned error can't find the container with id 5e4a7ba40ee614d96442e4fd5542c25e491b00ddce0f26c55ce8cf685a5ea0a1 Nov 02 21:53:36.972363 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:36.972343 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:53:37.546772 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:37.546729 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" event={"ID":"71a6aa31-bb91-41cd-a91b-d83d613e2275","Type":"ContainerStarted","Data":"5e4a7ba40ee614d96442e4fd5542c25e491b00ddce0f26c55ce8cf685a5ea0a1"} Nov 02 21:53:38.555474 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:38.555429 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" event={"ID":"71a6aa31-bb91-41cd-a91b-d83d613e2275","Type":"ContainerStarted","Data":"8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd"} Nov 02 21:53:38.556000 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:38.555543 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:38.684408 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:38.684352 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" podStartSLOduration=2.775637679 podStartE2EDuration="3.684333564s" totalImagesPullingTime="908.695885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:53:38.68364201 +0000 UTC m=+2444.659939785" watchObservedRunningTime="2026-11-02 21:53:38.684333564 +0000 UTC m=+2444.660631368" Nov 02 21:53:41.860855 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:41.860826 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:53:42.897384 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:42.897348 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9"] Nov 02 21:53:44.567966 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:44.567935 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 21:53:45.532216 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:45.532175 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:53:49.897940 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:49.897901 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j"] Nov 02 21:53:54.898397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:54.898355 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" podUID="d44b2c19-9cac-4549-bbbb-414729843150" containerName="payload-pre-processing" containerID="cri-o://df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082" gracePeriod=25 Nov 02 21:53:55.342861 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:55.342777 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" podUID="d44b2c19-9cac-4549-bbbb-414729843150" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:53:57.918149 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:57.918094 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" containerID="cri-o://da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0" gracePeriod=45 Nov 02 21:53:58.203253 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.203227 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:53:58.280214 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280187 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-home\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-home\") pod \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " Nov 02 21:53:58.280397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280313 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tmp-dir\") pod \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " Nov 02 21:53:58.280397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280347 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552a7878-b45f-4a9d-a82d-90618e9ac7a9-kube-api-access-2jhlw\" (UniqueName: \"kubernetes.io/projected/552a7878-b45f-4a9d-a82d-90618e9ac7a9-kube-api-access-2jhlw\") pod \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " Nov 02 21:53:58.280397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280373 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-model-cache\") pod \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " Nov 02 21:53:58.280619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280440 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-home" pod "552a7878-b45f-4a9d-a82d-90618e9ac7a9" (UID: "552a7878-b45f-4a9d-a82d-90618e9ac7a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.280619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280474 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-dshm\") pod \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " Nov 02 21:53:58.280619 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280509 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tls-certs\") pod \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\" (UID: \"552a7878-b45f-4a9d-a82d-90618e9ac7a9\") " Nov 02 21:53:58.280762 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280660 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tmp-dir" pod "552a7878-b45f-4a9d-a82d-90618e9ac7a9" (UID: "552a7878-b45f-4a9d-a82d-90618e9ac7a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.280879 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.280857 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-model-cache" pod "552a7878-b45f-4a9d-a82d-90618e9ac7a9" (UID: "552a7878-b45f-4a9d-a82d-90618e9ac7a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.281187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.281163 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.281187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.281189 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.281393 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.281206 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.282808 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.282786 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tls-certs" pod "552a7878-b45f-4a9d-a82d-90618e9ac7a9" (UID: "552a7878-b45f-4a9d-a82d-90618e9ac7a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:53:58.282911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.282863 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552a7878-b45f-4a9d-a82d-90618e9ac7a9-kube-api-access-2jhlw" pod "552a7878-b45f-4a9d-a82d-90618e9ac7a9" (UID: "552a7878-b45f-4a9d-a82d-90618e9ac7a9"). InnerVolumeSpecName "kube-api-access-2jhlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:53:58.282911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.282887 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-dshm" pod "552a7878-b45f-4a9d-a82d-90618e9ac7a9" (UID: "552a7878-b45f-4a9d-a82d-90618e9ac7a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:53:58.382114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.382082 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jhlw\" (UniqueName: \"kubernetes.io/projected/552a7878-b45f-4a9d-a82d-90618e9ac7a9-kube-api-access-2jhlw\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.382114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.382112 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552a7878-b45f-4a9d-a82d-90618e9ac7a9-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.382114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.382122 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552a7878-b45f-4a9d-a82d-90618e9ac7a9-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:53:58.683652 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.683620 4510 generic.go:356] "Generic (PLEG): container finished" podID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerID="da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0" exitCode=0 Nov 02 21:53:58.683824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.683663 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" Nov 02 21:53:58.683824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.683710 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" event={"ID":"552a7878-b45f-4a9d-a82d-90618e9ac7a9","Type":"ContainerDied","Data":"da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0"} Nov 02 21:53:58.683824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.683749 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" event={"ID":"552a7878-b45f-4a9d-a82d-90618e9ac7a9","Type":"ContainerDied","Data":"7fa4b25ed05cd06ad1e99a83c551a83b6947657627e0413160ac619c2ea75b73"} Nov 02 21:53:58.683824 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.683766 4510 scope.go:122] "RemoveContainer" containerID="da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0" Nov 02 21:53:58.695196 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.695179 4510 scope.go:122] "RemoveContainer" containerID="da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0" Nov 02 21:53:58.695451 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:58.695433 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0\": container with ID starting with da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0 not found: ID does not exist" containerID="da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0" Nov 02 21:53:58.695511 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.695464 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0"} err="failed to get container status \"da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0\": rpc error: code = NotFound desc = could not find container \"da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0\": container with ID starting with da0123ec2fb7fda88a039d81f4e677a889de4a0effd9abe0850f5ace953432b0 not found: ID does not exist" Nov 02 21:53:58.836617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.836584 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9"] Nov 02 21:53:58.882856 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:58.882824 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9"] Nov 02 21:53:59.155391 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.155349 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ef6b43/distinct-w5-ef6b43-kserve-79dfc856b6-k9jz9" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 21:53:59.432546 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.432442 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84"] Nov 02 21:53:59.433218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.433197 4510 cpu_manager.go:414] "containerMap: removing container" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" Nov 02 21:53:59.433218 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.433216 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" Nov 02 21:53:59.433560 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.433545 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" containerName="main" Nov 02 21:53:59.438788 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.438766 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.499802 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.499774 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84"] Nov 02 21:53:59.513867 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.513842 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-70d237-serving-cert\"" Nov 02 21:53:59.596776 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.596742 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d7f8-ea78-4117-86f8-2187e5724ba7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.596920 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.596789 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.596977 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.596926 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-metrics-certs\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.597024 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.596977 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cf82\" (UniqueName: \"kubernetes.io/projected/1c41d7f8-ea78-4117-86f8-2187e5724ba7-kube-api-access-5cf82\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.698602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.698465 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d7f8-ea78-4117-86f8-2187e5724ba7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.698602 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.698513 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.698863 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.698624 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-metrics-certs\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.698863 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.698691 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cf82\" (UniqueName: \"kubernetes.io/projected/1c41d7f8-ea78-4117-86f8-2187e5724ba7-kube-api-access-5cf82\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.699251 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.699231 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d7f8-ea78-4117-86f8-2187e5724ba7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.699344 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:59.699322 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-serving-cert: secret "maas-api-e2e-inf-b-70d237-serving-cert" not found Nov 02 21:53:59.699399 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:53:59.699371 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls podName:1c41d7f8-ea78-4117-86f8-2187e5724ba7 nodeName:}" failed. No retries permitted until 2026-11-02 21:54:00.199355982 +0000 UTC m=+2466.175653739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls") pod "maas-api-e2e-inf-b-70d237-788446b84d-6hw84" (UID: "1c41d7f8-ea78-4117-86f8-2187e5724ba7") : secret "maas-api-e2e-inf-b-70d237-serving-cert" not found Nov 02 21:53:59.701893 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.701871 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-metrics-certs\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:53:59.756720 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:53:59.756691 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cf82\" (UniqueName: \"kubernetes.io/projected/1c41d7f8-ea78-4117-86f8-2187e5724ba7-kube-api-access-5cf82\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:54:00.156019 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:00.155980 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552a7878-b45f-4a9d-a82d-90618e9ac7a9" path="/var/lib/kubelet/pods/552a7878-b45f-4a9d-a82d-90618e9ac7a9/volumes" Nov 02 21:54:00.206114 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:00.206086 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:54:00.208508 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:00.208477 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls\") pod \"maas-api-e2e-inf-b-70d237-788446b84d-6hw84\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:54:00.352507 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:00.352474 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:54:00.730648 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:00.730573 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84"] Nov 02 21:54:00.737269 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:54:00.737245 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c41d7f8_ea78_4117_86f8_2187e5724ba7.slice/crio-6fcd326c3b8d90286a0dfd80deb074566707ebbf0c2d1c1fed3ab3921a79cf2d WatchSource:0}: Error finding container 6fcd326c3b8d90286a0dfd80deb074566707ebbf0c2d1c1fed3ab3921a79cf2d: Status 404 returned error can't find the container with id 6fcd326c3b8d90286a0dfd80deb074566707ebbf0c2d1c1fed3ab3921a79cf2d Nov 02 21:54:01.707976 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:01.707942 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" event={"ID":"1c41d7f8-ea78-4117-86f8-2187e5724ba7","Type":"ContainerStarted","Data":"c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8"} Nov 02 21:54:01.708372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:01.708045 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:54:01.708372 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:01.708074 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" event={"ID":"1c41d7f8-ea78-4117-86f8-2187e5724ba7","Type":"ContainerStarted","Data":"6fcd326c3b8d90286a0dfd80deb074566707ebbf0c2d1c1fed3ab3921a79cf2d"} Nov 02 21:54:01.843571 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:01.843500 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" podStartSLOduration=1.9715718899999999 podStartE2EDuration="2.843486633s" totalImagesPullingTime="871.914743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:01.841846896 +0000 UTC m=+2467.818144670" watchObservedRunningTime="2026-11-02 21:54:01.843486633 +0000 UTC m=+2467.819784409" Nov 02 21:54:05.342662 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:05.342626 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" podUID="d44b2c19-9cac-4549-bbbb-414729843150" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 21:54:07.720550 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:07.720506 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 21:54:15.173587 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.173513 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:54:15.298175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.298147 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44b2c19-9cac-4549-bbbb-414729843150-kube-api-access-s5wnn\" (UniqueName: \"kubernetes.io/projected/d44b2c19-9cac-4549-bbbb-414729843150-kube-api-access-s5wnn\") pod \"d44b2c19-9cac-4549-bbbb-414729843150\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " Nov 02 21:54:15.298330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.298291 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44b2c19-9cac-4549-bbbb-414729843150-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b2c19-9cac-4549-bbbb-414729843150-plugins-config-volume\") pod \"d44b2c19-9cac-4549-bbbb-414729843150\" (UID: \"d44b2c19-9cac-4549-bbbb-414729843150\") " Nov 02 21:54:15.298757 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.298725 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44b2c19-9cac-4549-bbbb-414729843150-plugins-config-volume" pod "d44b2c19-9cac-4549-bbbb-414729843150" (UID: "d44b2c19-9cac-4549-bbbb-414729843150"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:15.299144 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.299118 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b2c19-9cac-4549-bbbb-414729843150-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:15.300467 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.300442 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44b2c19-9cac-4549-bbbb-414729843150-kube-api-access-s5wnn" pod "d44b2c19-9cac-4549-bbbb-414729843150" (UID: "d44b2c19-9cac-4549-bbbb-414729843150"). InnerVolumeSpecName "kube-api-access-s5wnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:15.400368 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.400340 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5wnn\" (UniqueName: \"kubernetes.io/projected/d44b2c19-9cac-4549-bbbb-414729843150-kube-api-access-s5wnn\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:15.532671 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:54:15.532590 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:54:15.816466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.816384 4510 generic.go:356] "Generic (PLEG): container finished" podID="d44b2c19-9cac-4549-bbbb-414729843150" containerID="df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082" exitCode=0 Nov 02 21:54:15.816466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.816426 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" Nov 02 21:54:15.816466 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.816447 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" event={"ID":"d44b2c19-9cac-4549-bbbb-414729843150","Type":"ContainerDied","Data":"df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082"} Nov 02 21:54:15.816751 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.816497 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j" event={"ID":"d44b2c19-9cac-4549-bbbb-414729843150","Type":"ContainerDied","Data":"1eae32f39bec983e2a27964d58806a0b7ff794aa3eb4d0196031ae7b2353d837"} Nov 02 21:54:15.816751 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.816554 4510 scope.go:122] "RemoveContainer" containerID="df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082" Nov 02 21:54:15.827371 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.827352 4510 scope.go:122] "RemoveContainer" containerID="df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082" Nov 02 21:54:15.827667 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:54:15.827648 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082\": container with ID starting with df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082 not found: ID does not exist" containerID="df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082" Nov 02 21:54:15.827715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.827675 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082"} err="failed to get container status \"df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082\": rpc error: code = NotFound desc = could not find container \"df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082\": container with ID starting with df1af12272d4674be6e50b7de8ed1168843abb01fb831899778eaf71fa804082 not found: ID does not exist" Nov 02 21:54:15.969815 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.969776 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j"] Nov 02 21:54:15.979470 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:15.979431 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ef6b43-fb6db9dc5-8kf8j"] Nov 02 21:54:16.155177 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:16.155143 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44b2c19-9cac-4549-bbbb-414729843150" path="/var/lib/kubelet/pods/d44b2c19-9cac-4549-bbbb-414729843150/volumes" Nov 02 21:54:22.839703 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:22.839659 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 21:54:22.840278 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:22.839735 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 21:54:45.553872 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:54:45.553821 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:54:51.187299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.187270 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44"] Nov 02 21:54:51.187969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.187954 4510 cpu_manager.go:414] "containerMap: removing container" podUID="d44b2c19-9cac-4549-bbbb-414729843150" containerName="payload-pre-processing" Nov 02 21:54:51.188017 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.187970 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44b2c19-9cac-4549-bbbb-414729843150" containerName="payload-pre-processing" Nov 02 21:54:51.188086 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.188077 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44b2c19-9cac-4549-bbbb-414729843150" containerName="payload-pre-processing" Nov 02 21:54:51.192108 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.192090 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.287831 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.287802 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptpzs\" (UniqueName: \"kubernetes.io/projected/d29a6515-d0a5-4982-b781-1459c27f6d8b-kube-api-access-ptpzs\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.288042 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.288016 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-metrics-certs\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.288123 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.288064 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a6515-d0a5-4982-b781-1459c27f6d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.288197 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.288178 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.290721 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.290700 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bcb857-serving-cert\"" Nov 02 21:54:51.294207 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.294186 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44"] Nov 02 21:54:51.389709 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.389674 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.389897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.389774 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptpzs\" (UniqueName: \"kubernetes.io/projected/d29a6515-d0a5-4982-b781-1459c27f6d8b-kube-api-access-ptpzs\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.389897 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:54:51.389848 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-serving-cert: secret "maas-api-e2e-inf-b-bcb857-serving-cert" not found Nov 02 21:54:51.389897 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.389892 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-metrics-certs\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.390095 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:54:51.389934 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls podName:d29a6515-d0a5-4982-b781-1459c27f6d8b nodeName:}" failed. No retries permitted until 2026-11-02 21:54:51.889909618 +0000 UTC m=+2517.866207398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls") pod "maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" (UID: "d29a6515-d0a5-4982-b781-1459c27f6d8b") : secret "maas-api-e2e-inf-b-bcb857-serving-cert" not found Nov 02 21:54:51.390095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.389958 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a6515-d0a5-4982-b781-1459c27f6d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.390573 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.390548 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a6515-d0a5-4982-b781-1459c27f6d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.392306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.392285 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-metrics-certs\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.450035 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.449952 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptpzs\" (UniqueName: \"kubernetes.io/projected/d29a6515-d0a5-4982-b781-1459c27f6d8b-kube-api-access-ptpzs\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.897826 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.897793 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:51.900126 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:51.900097 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:52.105617 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:52.105588 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:52.486440 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:52.486350 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44"] Nov 02 21:54:52.492663 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:54:52.492624 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29a6515_d0a5_4982_b781_1459c27f6d8b.slice/crio-dc93ab6a96eba0f2d774a67a798ba257af0bab1fb3d218c55005408374c71fdf WatchSource:0}: Error finding container dc93ab6a96eba0f2d774a67a798ba257af0bab1fb3d218c55005408374c71fdf: Status 404 returned error can't find the container with id dc93ab6a96eba0f2d774a67a798ba257af0bab1fb3d218c55005408374c71fdf Nov 02 21:54:53.076958 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:53.076924 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" event={"ID":"d29a6515-d0a5-4982-b781-1459c27f6d8b","Type":"ContainerStarted","Data":"dc93ab6a96eba0f2d774a67a798ba257af0bab1fb3d218c55005408374c71fdf"} Nov 02 21:54:53.133773 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:53.133744 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp"] Nov 02 21:54:53.225926 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:53.225896 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg"] Nov 02 21:54:54.087564 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:54.085949 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" event={"ID":"d29a6515-d0a5-4982-b781-1459c27f6d8b","Type":"ContainerStarted","Data":"989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e"} Nov 02 21:54:54.088207 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:54.086067 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:54:54.197360 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:54.197303 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" podStartSLOduration=2.38604667 podStartE2EDuration="3.197288462s" totalImagesPullingTime="811.241792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:54:54.193753959 +0000 UTC m=+2520.170051748" watchObservedRunningTime="2026-11-02 21:54:54.197288462 +0000 UTC m=+2520.173586236" Nov 02 21:54:58.134203 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.134166 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" podUID="2b9b86b0-9191-4fb3-87fb-944da476a3c4" containerName="payload-processing" containerID="cri-o://0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c" gracePeriod=25 Nov 02 21:54:58.226301 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.226261 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" podUID="08ad13ea-3381-4aaf-8547-9e16c042420c" containerName="payload-pre-processing" containerID="cri-o://546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99" gracePeriod=25 Nov 02 21:54:58.506390 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.506359 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:54:58.537354 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.537332 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:54:58.581396 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.581371 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ad13ea-3381-4aaf-8547-9e16c042420c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ad13ea-3381-4aaf-8547-9e16c042420c-plugins-config-volume\") pod \"08ad13ea-3381-4aaf-8547-9e16c042420c\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " Nov 02 21:54:58.581507 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.581439 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9b86b0-9191-4fb3-87fb-944da476a3c4-kube-api-access-gcr5k\" (UniqueName: \"kubernetes.io/projected/2b9b86b0-9191-4fb3-87fb-944da476a3c4-kube-api-access-gcr5k\") pod \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " Nov 02 21:54:58.581507 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.581498 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ad13ea-3381-4aaf-8547-9e16c042420c-kube-api-access-b4nbs\" (UniqueName: \"kubernetes.io/projected/08ad13ea-3381-4aaf-8547-9e16c042420c-kube-api-access-b4nbs\") pod \"08ad13ea-3381-4aaf-8547-9e16c042420c\" (UID: \"08ad13ea-3381-4aaf-8547-9e16c042420c\") " Nov 02 21:54:58.581738 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.581718 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9b86b0-9191-4fb3-87fb-944da476a3c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9b86b0-9191-4fb3-87fb-944da476a3c4-plugins-config-volume\") pod \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\" (UID: \"2b9b86b0-9191-4fb3-87fb-944da476a3c4\") " Nov 02 21:54:58.581789 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.581729 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ad13ea-3381-4aaf-8547-9e16c042420c-plugins-config-volume" pod "08ad13ea-3381-4aaf-8547-9e16c042420c" (UID: "08ad13ea-3381-4aaf-8547-9e16c042420c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:58.582079 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.582056 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9b86b0-9191-4fb3-87fb-944da476a3c4-plugins-config-volume" pod "2b9b86b0-9191-4fb3-87fb-944da476a3c4" (UID: "2b9b86b0-9191-4fb3-87fb-944da476a3c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:54:58.582243 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.582227 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9b86b0-9191-4fb3-87fb-944da476a3c4-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:58.582314 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.582245 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ad13ea-3381-4aaf-8547-9e16c042420c-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:58.583489 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.583467 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9b86b0-9191-4fb3-87fb-944da476a3c4-kube-api-access-gcr5k" pod "2b9b86b0-9191-4fb3-87fb-944da476a3c4" (UID: "2b9b86b0-9191-4fb3-87fb-944da476a3c4"). InnerVolumeSpecName "kube-api-access-gcr5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:58.583588 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.583513 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ad13ea-3381-4aaf-8547-9e16c042420c-kube-api-access-b4nbs" pod "08ad13ea-3381-4aaf-8547-9e16c042420c" (UID: "08ad13ea-3381-4aaf-8547-9e16c042420c"). InnerVolumeSpecName "kube-api-access-b4nbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:54:58.683515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.683446 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcr5k\" (UniqueName: \"kubernetes.io/projected/2b9b86b0-9191-4fb3-87fb-944da476a3c4-kube-api-access-gcr5k\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:58.683515 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:58.683473 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4nbs\" (UniqueName: \"kubernetes.io/projected/08ad13ea-3381-4aaf-8547-9e16c042420c-kube-api-access-b4nbs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:54:59.117670 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.117635 4510 generic.go:356] "Generic (PLEG): container finished" podID="08ad13ea-3381-4aaf-8547-9e16c042420c" containerID="546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99" exitCode=0 Nov 02 21:54:59.117862 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.117686 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" Nov 02 21:54:59.117862 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.117716 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" event={"ID":"08ad13ea-3381-4aaf-8547-9e16c042420c","Type":"ContainerDied","Data":"546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99"} Nov 02 21:54:59.117862 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.117770 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg" event={"ID":"08ad13ea-3381-4aaf-8547-9e16c042420c","Type":"ContainerDied","Data":"fbd0047603e78eff2df9ea2b946f278af3379ff2faf20e4994943fde1e77522e"} Nov 02 21:54:59.117862 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.117801 4510 scope.go:122] "RemoveContainer" containerID="546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99" Nov 02 21:54:59.119344 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.119317 4510 generic.go:356] "Generic (PLEG): container finished" podID="2b9b86b0-9191-4fb3-87fb-944da476a3c4" containerID="0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c" exitCode=0 Nov 02 21:54:59.119486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.119375 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" Nov 02 21:54:59.119486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.119383 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" event={"ID":"2b9b86b0-9191-4fb3-87fb-944da476a3c4","Type":"ContainerDied","Data":"0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c"} Nov 02 21:54:59.119486 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.119421 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp" event={"ID":"2b9b86b0-9191-4fb3-87fb-944da476a3c4","Type":"ContainerDied","Data":"c20c73fd94098e66d04a57f48e72b28dddffbaffaef43d2556a8fd5f917f5c6c"} Nov 02 21:54:59.135025 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.134886 4510 scope.go:122] "RemoveContainer" containerID="546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99" Nov 02 21:54:59.135300 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:54:59.135211 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99\": container with ID starting with 546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99 not found: ID does not exist" containerID="546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99" Nov 02 21:54:59.135300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.135240 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99"} err="failed to get container status \"546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99\": rpc error: code = NotFound desc = could not find container \"546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99\": container with ID starting with 546c201adb72208cd610951dd4a34ac50018b2210c791fd9d4a0f6ac9b3d9c99 not found: ID does not exist" Nov 02 21:54:59.135300 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.135256 4510 scope.go:122] "RemoveContainer" containerID="0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c" Nov 02 21:54:59.145807 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.145776 4510 scope.go:122] "RemoveContainer" containerID="0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c" Nov 02 21:54:59.146097 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:54:59.146072 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c\": container with ID starting with 0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c not found: ID does not exist" containerID="0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c" Nov 02 21:54:59.146189 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.146117 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c"} err="failed to get container status \"0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c\": rpc error: code = NotFound desc = could not find container \"0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c\": container with ID starting with 0a0c64ab626dde183a51f6a49c6f51a390ea7f5c560e8ba39965bb314064296c not found: ID does not exist" Nov 02 21:54:59.270595 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.270567 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg"] Nov 02 21:54:59.273958 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.273928 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d0946d-8465488fff-wnnmg"] Nov 02 21:54:59.373921 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.373847 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp"] Nov 02 21:54:59.377327 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.377302 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d0946d-59f4d669d9-k4wgp"] Nov 02 21:54:59.898559 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:54:59.898508 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:55:00.154287 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:55:00.154207 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ad13ea-3381-4aaf-8547-9e16c042420c" path="/var/lib/kubelet/pods/08ad13ea-3381-4aaf-8547-9e16c042420c/volumes" Nov 02 21:55:00.154810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:55:00.154792 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9b86b0-9191-4fb3-87fb-944da476a3c4" path="/var/lib/kubelet/pods/2b9b86b0-9191-4fb3-87fb-944da476a3c4/volumes" Nov 02 21:55:01.105723 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:55:01.105696 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 21:55:15.531345 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:55:15.531307 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:55:44.990031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:55:44.989949 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:55:45.545199 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:55:45.545161 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:56:15.531821 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:56:15.531788 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:56:45.532715 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:56:45.532675 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:56:52.550386 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:52.550349 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:56:56.905190 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.905157 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc"] Nov 02 21:56:56.905885 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.905867 4510 cpu_manager.go:414] "containerMap: removing container" podUID="2b9b86b0-9191-4fb3-87fb-944da476a3c4" containerName="payload-processing" Nov 02 21:56:56.905943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.905885 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9b86b0-9191-4fb3-87fb-944da476a3c4" containerName="payload-processing" Nov 02 21:56:56.905943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.905916 4510 cpu_manager.go:414] "containerMap: removing container" podUID="08ad13ea-3381-4aaf-8547-9e16c042420c" containerName="payload-pre-processing" Nov 02 21:56:56.905943 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.905922 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ad13ea-3381-4aaf-8547-9e16c042420c" containerName="payload-pre-processing" Nov 02 21:56:56.906045 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.906017 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9b86b0-9191-4fb3-87fb-944da476a3c4" containerName="payload-processing" Nov 02 21:56:56.906045 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.906032 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ad13ea-3381-4aaf-8547-9e16c042420c" containerName="payload-pre-processing" Nov 02 21:56:56.909715 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.909698 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:56.959726 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.959696 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3566f9\"/\"openshift-service-ca.crt\"" Nov 02 21:56:56.974692 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.974659 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tls-certs\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:56.974903 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.974878 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-model-cache\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:56.974969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.974919 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dknd9\" (UniqueName: \"kubernetes.io/projected/b8b15b7e-e88e-47a2-9472-a26694c11ce9-kube-api-access-dknd9\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:56.974969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.974936 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-home\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:56.975105 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.975083 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-dshm\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:56.975146 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:56.975116 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tmp-dir\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.005882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.005851 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3566f9\"/\"default-dockercfg-kmjpg\"" Nov 02 21:56:57.005882 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.005862 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3566f9\"/\"test-model-3566f9-kserve-self-signed-certs\"" Nov 02 21:56:57.006066 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.005903 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3566f9\"/\"kube-root-ca.crt\"" Nov 02 21:56:57.009262 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.009242 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc"] Nov 02 21:56:57.076291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076257 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-dshm\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.076291 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076293 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tmp-dir\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.076485 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076425 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tls-certs\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.076643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076626 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-model-cache\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.076747 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076670 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dknd9\" (UniqueName: \"kubernetes.io/projected/b8b15b7e-e88e-47a2-9472-a26694c11ce9-kube-api-access-dknd9\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.076842 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076776 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-home\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.076911 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076770 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tmp-dir\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.076969 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.076928 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-model-cache\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.077031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.077009 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-home\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.078627 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.078607 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-dshm\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.078944 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.078928 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tls-certs\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.129234 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.129200 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dknd9\" (UniqueName: \"kubernetes.io/projected/b8b15b7e-e88e-47a2-9472-a26694c11ce9-kube-api-access-dknd9\") pod \"test-model-3566f9-kserve-55486df8c7-t7qpc\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.222378 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.222308 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:56:57.594496 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.594420 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc"] Nov 02 21:56:57.600829 ip-10-0-133-33 kubenswrapper[4510]: W1102 21:56:57.600801 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8b15b7e_e88e_47a2_9472_a26694c11ce9.slice/crio-adcdc51f0cf0cd6b84b2abea67b568a5229a7df3c34defe39bedf89da2709c75 WatchSource:0}: Error finding container adcdc51f0cf0cd6b84b2abea67b568a5229a7df3c34defe39bedf89da2709c75: Status 404 returned error can't find the container with id adcdc51f0cf0cd6b84b2abea67b568a5229a7df3c34defe39bedf89da2709c75 Nov 02 21:56:57.882268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.882232 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" event={"ID":"b8b15b7e-e88e-47a2-9472-a26694c11ce9","Type":"ContainerStarted","Data":"360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb"} Nov 02 21:56:57.882427 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.882281 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" event={"ID":"b8b15b7e-e88e-47a2-9472-a26694c11ce9","Type":"ContainerStarted","Data":"adcdc51f0cf0cd6b84b2abea67b568a5229a7df3c34defe39bedf89da2709c75"} Nov 02 21:56:57.990335 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:57.990281 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" podStartSLOduration=1.990268701 podStartE2EDuration="1.990268701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:56:57.988469954 +0000 UTC m=+2643.964767729" watchObservedRunningTime="2026-11-02 21:56:57.990268701 +0000 UTC m=+2643.966566475" Nov 02 21:56:58.223456 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:56:58.223381 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:57:05.381037 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.381000 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs"] Nov 02 21:57:05.381545 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.381311 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" podUID="4f5f9f36-7d12-4dab-b335-3cbb45768e62" containerName="maas-api" containerID="cri-o://6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f" gracePeriod=30 Nov 02 21:57:05.660031 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.660006 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:57:05.789488 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.789454 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5f9f36-7d12-4dab-b335-3cbb45768e62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5f9f36-7d12-4dab-b335-3cbb45768e62-openshift-service-ca\") pod \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " Nov 02 21:57:05.789710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.789584 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls\") pod \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " Nov 02 21:57:05.789710 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.789620 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5f9f36-7d12-4dab-b335-3cbb45768e62-kube-api-access-k9r8m\" (UniqueName: \"kubernetes.io/projected/4f5f9f36-7d12-4dab-b335-3cbb45768e62-kube-api-access-k9r8m\") pod \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " Nov 02 21:57:05.789857 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.789834 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-metrics-certs\") pod \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\" (UID: \"4f5f9f36-7d12-4dab-b335-3cbb45768e62\") " Nov 02 21:57:05.790739 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.789883 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5f9f36-7d12-4dab-b335-3cbb45768e62-openshift-service-ca" pod "4f5f9f36-7d12-4dab-b335-3cbb45768e62" (UID: "4f5f9f36-7d12-4dab-b335-3cbb45768e62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:57:05.792416 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.792387 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5f9f36-7d12-4dab-b335-3cbb45768e62-kube-api-access-k9r8m" pod "4f5f9f36-7d12-4dab-b335-3cbb45768e62" (UID: "4f5f9f36-7d12-4dab-b335-3cbb45768e62"). InnerVolumeSpecName "kube-api-access-k9r8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:57:05.792652 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.792625 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls" pod "4f5f9f36-7d12-4dab-b335-3cbb45768e62" (UID: "4f5f9f36-7d12-4dab-b335-3cbb45768e62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:05.801958 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.801930 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-metrics-certs" pod "4f5f9f36-7d12-4dab-b335-3cbb45768e62" (UID: "4f5f9f36-7d12-4dab-b335-3cbb45768e62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:57:05.892463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.892389 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5f9f36-7d12-4dab-b335-3cbb45768e62-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:05.892463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.892416 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:05.892463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.892426 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9r8m\" (UniqueName: \"kubernetes.io/projected/4f5f9f36-7d12-4dab-b335-3cbb45768e62-kube-api-access-k9r8m\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:05.892463 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.892435 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f5f9f36-7d12-4dab-b335-3cbb45768e62-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:57:05.934277 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.934245 4510 generic.go:356] "Generic (PLEG): container finished" podID="4f5f9f36-7d12-4dab-b335-3cbb45768e62" containerID="6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f" exitCode=0 Nov 02 21:57:05.934439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.934293 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" Nov 02 21:57:05.934439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.934301 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" event={"ID":"4f5f9f36-7d12-4dab-b335-3cbb45768e62","Type":"ContainerDied","Data":"6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f"} Nov 02 21:57:05.934439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.934341 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs" event={"ID":"4f5f9f36-7d12-4dab-b335-3cbb45768e62","Type":"ContainerDied","Data":"3bd33348221ceab30366860a678ffbe319aefb4da58b031bec0dc2fbf9b7cff8"} Nov 02 21:57:05.934439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.934361 4510 scope.go:122] "RemoveContainer" containerID="6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f" Nov 02 21:57:05.945311 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.945291 4510 scope.go:122] "RemoveContainer" containerID="6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f" Nov 02 21:57:05.945604 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:57:05.945586 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f\": container with ID starting with 6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f not found: ID does not exist" containerID="6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f" Nov 02 21:57:05.945652 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:05.945615 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f"} err="failed to get container status \"6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f\": rpc error: code = NotFound desc = could not find container \"6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f\": container with ID starting with 6df58998002be8143806bfdeb0875cccc8b39c98283ae44bc961403436e4d13f not found: ID does not exist" Nov 02 21:57:06.087038 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:06.087001 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs"] Nov 02 21:57:06.090332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:06.090302 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d0946d-5cf8669d9d-25pxs"] Nov 02 21:57:06.155269 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:06.155202 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5f9f36-7d12-4dab-b335-3cbb45768e62" path="/var/lib/kubelet/pods/4f5f9f36-7d12-4dab-b335-3cbb45768e62/volumes" Nov 02 21:57:07.223293 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:07.223258 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:57:07.235654 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:07.235631 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:57:07.958846 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:07.958817 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 21:57:15.531639 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:57:15.531599 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:57:45.531152 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:57:45.531112 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:57:59.909687 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:57:59.909652 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:58:15.532773 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:58:15.532735 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:58:41.433370 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:41.433292 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h"] Nov 02 21:58:45.531083 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:58:45.531038 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:58:56.454229 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.454171 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" podUID="fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" containerName="main" containerID="cri-o://0366bf2a983e3ef3023989ede52cc353189f63a07cb27a8b01036bd4bcc085f6" gracePeriod=45 Nov 02 21:58:56.468837 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.468807 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" podUID="fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.134:8000/ready\": dial tcp 10.134.0.134:8000: connect: connection refused" Nov 02 21:58:56.645889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.645851 4510 generic.go:356] "Generic (PLEG): container finished" podID="fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" containerID="0366bf2a983e3ef3023989ede52cc353189f63a07cb27a8b01036bd4bcc085f6" exitCode=0 Nov 02 21:58:56.646029 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.645926 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" event={"ID":"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6","Type":"ContainerDied","Data":"0366bf2a983e3ef3023989ede52cc353189f63a07cb27a8b01036bd4bcc085f6"} Nov 02 21:58:56.735392 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.735369 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:58:56.875290 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875258 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-dshm\") pod \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " Nov 02 21:58:56.875455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875352 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-kube-api-access-mlz5w\" (UniqueName: \"kubernetes.io/projected/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-kube-api-access-mlz5w\") pod \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " Nov 02 21:58:56.875455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875385 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tls-certs\") pod \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " Nov 02 21:58:56.875455 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875418 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-home\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-home\") pod \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " Nov 02 21:58:56.875659 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875617 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-model-cache\") pod \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " Nov 02 21:58:56.875722 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875682 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tmp-dir\") pod \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\" (UID: \"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6\") " Nov 02 21:58:56.876071 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875917 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-home" pod "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" (UID: "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:56.876071 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.875958 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-model-cache" pod "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" (UID: "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:56.876071 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.876044 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tmp-dir" pod "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" (UID: "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:56.876322 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.876304 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:56.876397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.876328 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:56.876397 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.876344 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:56.878175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.878143 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-kube-api-access-mlz5w" pod "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" (UID: "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6"). InnerVolumeSpecName "kube-api-access-mlz5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:56.878175 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.878148 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-dshm" pod "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" (UID: "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:56.878352 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.878198 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tls-certs" pod "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" (UID: "fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:56.977268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.977193 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:56.977268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.977222 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlz5w\" (UniqueName: \"kubernetes.io/projected/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-kube-api-access-mlz5w\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:56.977268 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:56.977239 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:57.654299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:57.654266 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" Nov 02 21:58:57.654299 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:57.654292 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h" event={"ID":"fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6","Type":"ContainerDied","Data":"9917eef8c644056250cd8e259df9872b99acb1c85552785ea00f0b14fe5134e3"} Nov 02 21:58:57.654736 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:57.654345 4510 scope.go:122] "RemoveContainer" containerID="0366bf2a983e3ef3023989ede52cc353189f63a07cb27a8b01036bd4bcc085f6" Nov 02 21:58:57.808137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:57.808110 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h"] Nov 02 21:58:57.854491 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:57.854458 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7cc1b1/test-model-7cc1b1-kserve-7c557d8449-kn42h"] Nov 02 21:58:58.155364 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:58:58.155330 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" path="/var/lib/kubelet/pods/fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6/volumes" Nov 02 21:59:15.531495 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:59:15.531460 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:59:19.838596 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:19.838556 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 21:59:45.533992 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:59:45.533951 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 21:59:46.784636 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:46.784591 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp"] Nov 02 21:59:46.785406 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:46.784872 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" podUID="3f233ae1-2af8-4a95-8393-8b9a074b3034" containerName="maas-api" containerID="cri-o://06a384921f52a781cbe9a74367e78ebb8131c783d02d9c51ce9a248211264fb4" gracePeriod=30 Nov 02 21:59:46.986691 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:46.986654 4510 generic.go:356] "Generic (PLEG): container finished" podID="3f233ae1-2af8-4a95-8393-8b9a074b3034" containerID="06a384921f52a781cbe9a74367e78ebb8131c783d02d9c51ce9a248211264fb4" exitCode=0 Nov 02 21:59:46.986887 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:46.986834 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" event={"ID":"3f233ae1-2af8-4a95-8393-8b9a074b3034","Type":"ContainerDied","Data":"06a384921f52a781cbe9a74367e78ebb8131c783d02d9c51ce9a248211264fb4"} Nov 02 21:59:47.062227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.062197 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:59:47.177593 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.177559 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls\") pod \"3f233ae1-2af8-4a95-8393-8b9a074b3034\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " Nov 02 21:59:47.177783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.177627 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f233ae1-2af8-4a95-8393-8b9a074b3034-kube-api-access-x5bt9\" (UniqueName: \"kubernetes.io/projected/3f233ae1-2af8-4a95-8393-8b9a074b3034-kube-api-access-x5bt9\") pod \"3f233ae1-2af8-4a95-8393-8b9a074b3034\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " Nov 02 21:59:47.177783 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.177709 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-metrics-certs\") pod \"3f233ae1-2af8-4a95-8393-8b9a074b3034\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " Nov 02 21:59:47.177933 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.177909 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f233ae1-2af8-4a95-8393-8b9a074b3034-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f233ae1-2af8-4a95-8393-8b9a074b3034-openshift-service-ca\") pod \"3f233ae1-2af8-4a95-8393-8b9a074b3034\" (UID: \"3f233ae1-2af8-4a95-8393-8b9a074b3034\") " Nov 02 21:59:47.178330 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.178298 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f233ae1-2af8-4a95-8393-8b9a074b3034-openshift-service-ca" pod "3f233ae1-2af8-4a95-8393-8b9a074b3034" (UID: "3f233ae1-2af8-4a95-8393-8b9a074b3034"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:47.178643 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.178615 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f233ae1-2af8-4a95-8393-8b9a074b3034-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.180309 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.180288 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-metrics-certs" pod "3f233ae1-2af8-4a95-8393-8b9a074b3034" (UID: "3f233ae1-2af8-4a95-8393-8b9a074b3034"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:47.180410 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.180336 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f233ae1-2af8-4a95-8393-8b9a074b3034-kube-api-access-x5bt9" pod "3f233ae1-2af8-4a95-8393-8b9a074b3034" (UID: "3f233ae1-2af8-4a95-8393-8b9a074b3034"). InnerVolumeSpecName "kube-api-access-x5bt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:47.180565 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.180511 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls" pod "3f233ae1-2af8-4a95-8393-8b9a074b3034" (UID: "3f233ae1-2af8-4a95-8393-8b9a074b3034"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:47.280247 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.280215 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.280247 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.280249 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5bt9\" (UniqueName: \"kubernetes.io/projected/3f233ae1-2af8-4a95-8393-8b9a074b3034-kube-api-access-x5bt9\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.280439 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.280266 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f233ae1-2af8-4a95-8393-8b9a074b3034-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:47.995156 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.995069 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" event={"ID":"3f233ae1-2af8-4a95-8393-8b9a074b3034","Type":"ContainerDied","Data":"4d7d8936518e75991ef204b7d42ea1b83b1ba8287518142581c890ca377c8960"} Nov 02 21:59:47.995156 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.995102 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp" Nov 02 21:59:47.995156 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:47.995129 4510 scope.go:122] "RemoveContainer" containerID="06a384921f52a781cbe9a74367e78ebb8131c783d02d9c51ce9a248211264fb4" Nov 02 21:59:48.155503 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:48.155473 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp"] Nov 02 21:59:48.193907 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:48.193878 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d2b29-7f46875cb8-s5hqp"] Nov 02 21:59:50.156244 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:50.156207 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f233ae1-2af8-4a95-8393-8b9a074b3034" path="/var/lib/kubelet/pods/3f233ae1-2af8-4a95-8393-8b9a074b3034/volumes" Nov 02 21:59:52.503227 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:52.503192 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k"] Nov 02 21:59:52.503745 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:52.503444 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" containerID="cri-o://0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac" gracePeriod=30 Nov 02 21:59:52.548997 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:52.548961 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc"] Nov 02 21:59:52.549187 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:52.549154 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" containerID="cri-o://c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2" gracePeriod=30 Nov 02 21:59:56.195834 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:56.195798 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:59:56.196267 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:56.195854 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:59:56.245813 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:56.245786 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 21:59:56.245953 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:56.245831 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 21:59:57.827323 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.827297 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:59:57.850306 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.850282 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:59:57.911201 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911174 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-podinfo\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.911332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911223 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-socket\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.911332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911240 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-certs\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.911332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911263 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-certs\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.911332 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911303 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-envoy\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.911502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911370 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-token\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.911502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911404 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-data\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.911502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911440 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-data\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.911502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911472 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-socket" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.911502 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911487 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istiod-ca-cert\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.911810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911572 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istiod-ca-cert\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.911810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911612 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-certs" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.911810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911634 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-certs" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.911810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911676 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-envoy\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.911810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911726 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-credential-socket\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.911810 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911756 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-kube-api-access-8bd5g\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-kube-api-access-8bd5g\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911794 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-data" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911828 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-data" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911831 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-token\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-token\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911924 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-credential-socket\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.911967 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-kube-api-access-s4vd2\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-kube-api-access-s4vd2\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.912007 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-podinfo\") pod \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\" (UID: \"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c\") " Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.912019 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-credential-socket" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.912028 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istiod-ca-cert" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:57.912137 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.912051 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istiod-ca-cert" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:57.912623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.912330 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-credential-socket" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.912623 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.912425 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-socket\") pod \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\" (UID: \"3e1009d2-97e7-46dc-8e4c-0c297b269aab\") " Nov 02 21:59:57.913071 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913046 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913078 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913093 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913106 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913120 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913133 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913145 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913159 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913173 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913171 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:57.913668 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.913409 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-socket" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.914095 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.914040 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-podinfo" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:59:57.914366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.914340 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-token" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:57.914586 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.914556 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-kube-api-access-8bd5g" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "kube-api-access-8bd5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:57.914586 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.914577 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-podinfo" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 21:59:57.914711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.914588 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-envoy" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.914711 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.914675 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-token" pod "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" (UID: "9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:57.915033 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.915017 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-envoy" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:57.915366 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:57.915353 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-kube-api-access-s4vd2" pod "3e1009d2-97e7-46dc-8e4c-0c297b269aab" (UID: "3e1009d2-97e7-46dc-8e4c-0c297b269aab"). InnerVolumeSpecName "kube-api-access-s4vd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:58.013975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.013943 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.013975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.013971 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.013975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.013980 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.013975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.013989 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.013975 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.013997 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.014387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.014007 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bd5g\" (UniqueName: \"kubernetes.io/projected/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-kube-api-access-8bd5g\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.014387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.014016 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.014387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.014025 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4vd2\" (UniqueName: \"kubernetes.io/projected/3e1009d2-97e7-46dc-8e4c-0c297b269aab-kube-api-access-s4vd2\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.014387 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.014034 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:58.061778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.061697 4510 generic.go:356] "Generic (PLEG): container finished" podID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerID="c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2" exitCode=0 Nov 02 21:59:58.061778 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.061757 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" Nov 02 21:59:58.061991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.061784 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" event={"ID":"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c","Type":"ContainerDied","Data":"c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2"} Nov 02 21:59:58.061991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.061838 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc" event={"ID":"9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c","Type":"ContainerDied","Data":"f8dc40ad8fe8dbbd2197c9467a4390caddae44181e7348164b1ac11ed3d21947"} Nov 02 21:59:58.061991 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.061854 4510 scope.go:122] "RemoveContainer" containerID="c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2" Nov 02 21:59:58.063425 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.063405 4510 generic.go:356] "Generic (PLEG): container finished" podID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerID="0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac" exitCode=0 Nov 02 21:59:58.063563 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.063453 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" Nov 02 21:59:58.063563 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.063552 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" event={"ID":"3e1009d2-97e7-46dc-8e4c-0c297b269aab","Type":"ContainerDied","Data":"0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac"} Nov 02 21:59:58.063702 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.063596 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k" event={"ID":"3e1009d2-97e7-46dc-8e4c-0c297b269aab","Type":"ContainerDied","Data":"8a7f62cc26986fe4ebe82c18731dc66eae2f599314af393442c5f232f5170a26"} Nov 02 21:59:58.077889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.075054 4510 scope.go:122] "RemoveContainer" containerID="c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2" Nov 02 21:59:58.077889 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:59:58.075349 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2\": container with ID starting with c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2 not found: ID does not exist" containerID="c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2" Nov 02 21:59:58.077889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.075375 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2"} err="failed to get container status \"c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2\": rpc error: code = NotFound desc = could not find container \"c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2\": container with ID starting with c5a09bf35dfc426a5c4bd51d2f65c8f5a43de519bb26efe176e02c688f752ea2 not found: ID does not exist" Nov 02 21:59:58.077889 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.075396 4510 scope.go:122] "RemoveContainer" containerID="0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac" Nov 02 21:59:58.101954 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.101928 4510 scope.go:122] "RemoveContainer" containerID="0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac" Nov 02 21:59:58.102325 ip-10-0-133-33 kubenswrapper[4510]: E1102 21:59:58.102297 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac\": container with ID starting with 0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac not found: ID does not exist" containerID="0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac" Nov 02 21:59:58.102405 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.102336 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac"} err="failed to get container status \"0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac\": rpc error: code = NotFound desc = could not find container \"0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac\": container with ID starting with 0be4f84073a7e38cd7c385081c550fc73bf1652cb3dff131f5c2a5f77bbd07ac not found: ID does not exist" Nov 02 21:59:58.218394 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.218294 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc"] Nov 02 21:59:58.225217 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.224538 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-mwdzc"] Nov 02 21:59:58.325141 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.325021 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k"] Nov 02 21:59:58.328924 ip-10-0-133-33 kubenswrapper[4510]: I1102 21:59:58.328883 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-pjj8k"] Nov 02 22:00:00.156136 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:00.156102 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" path="/var/lib/kubelet/pods/3e1009d2-97e7-46dc-8e4c-0c297b269aab/volumes" Nov 02 22:00:00.156608 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:00.156592 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" path="/var/lib/kubelet/pods/9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c/volumes" Nov 02 22:00:15.530884 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:00:15.530801 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:00:21.121040 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:21.121006 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:00:43.336820 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.336786 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl"] Nov 02 22:00:43.337312 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.336977 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" podUID="40705686-3e8f-4f9c-8bc4-a647a83827b3" containerName="maas-api" containerID="cri-o://9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb" gracePeriod=30 Nov 02 22:00:43.619613 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.619589 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 22:00:43.769433 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.769404 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-metrics-certs\") pod \"40705686-3e8f-4f9c-8bc4-a647a83827b3\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " Nov 02 22:00:43.769666 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.769488 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40705686-3e8f-4f9c-8bc4-a647a83827b3-kube-api-access-s92fk\" (UniqueName: \"kubernetes.io/projected/40705686-3e8f-4f9c-8bc4-a647a83827b3-kube-api-access-s92fk\") pod \"40705686-3e8f-4f9c-8bc4-a647a83827b3\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " Nov 02 22:00:43.769835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.769769 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") pod \"40705686-3e8f-4f9c-8bc4-a647a83827b3\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " Nov 02 22:00:43.769835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.769820 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40705686-3e8f-4f9c-8bc4-a647a83827b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40705686-3e8f-4f9c-8bc4-a647a83827b3-openshift-service-ca\") pod \"40705686-3e8f-4f9c-8bc4-a647a83827b3\" (UID: \"40705686-3e8f-4f9c-8bc4-a647a83827b3\") " Nov 02 22:00:43.770387 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.770360 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40705686-3e8f-4f9c-8bc4-a647a83827b3-openshift-service-ca" pod "40705686-3e8f-4f9c-8bc4-a647a83827b3" (UID: "40705686-3e8f-4f9c-8bc4-a647a83827b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:00:43.772100 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.772070 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-metrics-certs" pod "40705686-3e8f-4f9c-8bc4-a647a83827b3" (UID: "40705686-3e8f-4f9c-8bc4-a647a83827b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:43.772273 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.772255 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40705686-3e8f-4f9c-8bc4-a647a83827b3-kube-api-access-s92fk" pod "40705686-3e8f-4f9c-8bc4-a647a83827b3" (UID: "40705686-3e8f-4f9c-8bc4-a647a83827b3"). InnerVolumeSpecName "kube-api-access-s92fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:00:43.772436 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.772411 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls" pod "40705686-3e8f-4f9c-8bc4-a647a83827b3" (UID: "40705686-3e8f-4f9c-8bc4-a647a83827b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:00:43.871146 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.871116 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:43.871146 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.871143 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40705686-3e8f-4f9c-8bc4-a647a83827b3-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:43.871336 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.871152 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40705686-3e8f-4f9c-8bc4-a647a83827b3-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:43.871336 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:43.871161 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s92fk\" (UniqueName: \"kubernetes.io/projected/40705686-3e8f-4f9c-8bc4-a647a83827b3-kube-api-access-s92fk\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:00:44.357728 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.357693 4510 generic.go:356] "Generic (PLEG): container finished" podID="40705686-3e8f-4f9c-8bc4-a647a83827b3" containerID="9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb" exitCode=0 Nov 02 22:00:44.358234 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.357742 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" Nov 02 22:00:44.358234 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.357782 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" event={"ID":"40705686-3e8f-4f9c-8bc4-a647a83827b3","Type":"ContainerDied","Data":"9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb"} Nov 02 22:00:44.358234 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.357836 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" event={"ID":"40705686-3e8f-4f9c-8bc4-a647a83827b3","Type":"ContainerDied","Data":"d3023a76837d109270f20e8e8e846a3704de3c58aeeeecd2a02daf750a1c8ed2"} Nov 02 22:00:44.358234 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.357863 4510 scope.go:122] "RemoveContainer" containerID="9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb" Nov 02 22:00:44.368003 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.367980 4510 scope.go:122] "RemoveContainer" containerID="9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb" Nov 02 22:00:44.368262 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:00:44.368245 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb\": container with ID starting with 9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb not found: ID does not exist" containerID="9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb" Nov 02 22:00:44.368306 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.368271 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb"} err="failed to get container status \"9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb\": rpc error: code = NotFound desc = could not find container \"9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb\": container with ID starting with 9a3783789a8a40cc2872ff087e3860be852e57278c1df21c764406cf46a15ecb not found: ID does not exist" Nov 02 22:00:44.510607 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.510543 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl"] Nov 02 22:00:44.556612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:44.556589 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl"] Nov 02 22:00:45.533874 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:00:45.533832 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:00:46.154712 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:46.154679 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40705686-3e8f-4f9c-8bc4-a647a83827b3" path="/var/lib/kubelet/pods/40705686-3e8f-4f9c-8bc4-a647a83827b3/volumes" Nov 02 22:00:46.552988 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:00:46.552898 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7cc1b1-745dd7f55f-9wndl" podUID="40705686-3e8f-4f9c-8bc4-a647a83827b3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.119:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:01:05.925564 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:05.925530 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:01:12.754014 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.753983 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v"] Nov 02 22:01:12.754645 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754626 4510 cpu_manager.go:414] "containerMap: removing container" podUID="40705686-3e8f-4f9c-8bc4-a647a83827b3" containerName="maas-api" Nov 02 22:01:12.754713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754647 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40705686-3e8f-4f9c-8bc4-a647a83827b3" containerName="maas-api" Nov 02 22:01:12.754713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754665 4510 cpu_manager.go:414] "containerMap: removing container" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" Nov 02 22:01:12.754713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754671 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" Nov 02 22:01:12.754713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754694 4510 cpu_manager.go:414] "containerMap: removing container" podUID="3f233ae1-2af8-4a95-8393-8b9a074b3034" containerName="maas-api" Nov 02 22:01:12.754713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754700 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f233ae1-2af8-4a95-8393-8b9a074b3034" containerName="maas-api" Nov 02 22:01:12.754713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754712 4510 cpu_manager.go:414] "containerMap: removing container" podUID="fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" containerName="main" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754718 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" containerName="main" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754731 4510 cpu_manager.go:414] "containerMap: removing container" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754736 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754745 4510 cpu_manager.go:414] "containerMap: removing container" podUID="4f5f9f36-7d12-4dab-b335-3cbb45768e62" containerName="maas-api" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754749 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5f9f36-7d12-4dab-b335-3cbb45768e62" containerName="maas-api" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754841 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde1f82a-a3e0-4d9e-8b0e-9b91e62083a6" containerName="main" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754850 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1009d2-97e7-46dc-8e4c-0c297b269aab" containerName="istio-proxy" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754858 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5f9f36-7d12-4dab-b335-3cbb45768e62" containerName="maas-api" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754873 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="40705686-3e8f-4f9c-8bc4-a647a83827b3" containerName="maas-api" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754881 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b54d4fe-b9a2-4a9a-9c5e-438a75c10e6c" containerName="istio-proxy" Nov 02 22:01:12.754897 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.754891 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f233ae1-2af8-4a95-8393-8b9a074b3034" containerName="maas-api" Nov 02 22:01:12.759795 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.759776 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:12.855198 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.855157 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c4a6b9-serving-cert\"" Nov 02 22:01:12.858501 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.858476 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v"] Nov 02 22:01:12.918090 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.918061 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-metrics-certs\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:12.918253 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.918183 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4xc\" (UniqueName: \"kubernetes.io/projected/93ae6126-729e-4525-8e03-d9cda4256a3f-kube-api-access-xc4xc\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:12.918253 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.918222 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae6126-729e-4525-8e03-d9cda4256a3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:12.918357 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:12.918341 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.019662 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.019575 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae6126-729e-4525-8e03-d9cda4256a3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.019829 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.019672 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.019829 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.019773 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-metrics-certs\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.019945 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.019842 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4xc\" (UniqueName: \"kubernetes.io/projected/93ae6126-729e-4525-8e03-d9cda4256a3f-kube-api-access-xc4xc\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.020274 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.020252 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae6126-729e-4525-8e03-d9cda4256a3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.022163 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.022138 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.022284 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.022163 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-metrics-certs\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.073707 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.073676 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4xc\" (UniqueName: \"kubernetes.io/projected/93ae6126-729e-4525-8e03-d9cda4256a3f-kube-api-access-xc4xc\") pod \"maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.374110 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.374085 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:13.746648 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.746565 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v"] Nov 02 22:01:13.753112 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:01:13.753084 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ae6126_729e_4525_8e03_d9cda4256a3f.slice/crio-7e8018f25fae2e9c0363c1c38c12f4c56ce81fa1525fc0a27868bf56de10ce28 WatchSource:0}: Error finding container 7e8018f25fae2e9c0363c1c38c12f4c56ce81fa1525fc0a27868bf56de10ce28: Status 404 returned error can't find the container with id 7e8018f25fae2e9c0363c1c38c12f4c56ce81fa1525fc0a27868bf56de10ce28 Nov 02 22:01:13.755265 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:13.755244 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:01:14.544387 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:14.544353 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" event={"ID":"93ae6126-729e-4525-8e03-d9cda4256a3f","Type":"ContainerStarted","Data":"7e8018f25fae2e9c0363c1c38c12f4c56ce81fa1525fc0a27868bf56de10ce28"} Nov 02 22:01:15.532791 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:01:15.532755 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:01:15.553483 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:15.553449 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" event={"ID":"93ae6126-729e-4525-8e03-d9cda4256a3f","Type":"ContainerStarted","Data":"edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7"} Nov 02 22:01:15.553650 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:15.553512 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:15.661629 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:15.661576 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" podStartSLOduration=2.730328298 podStartE2EDuration="3.661562805s" totalImagesPullingTime="931.234507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:01:15.659290488 +0000 UTC m=+2901.635588263" watchObservedRunningTime="2026-11-02 22:01:15.661562805 +0000 UTC m=+2901.637860580" Nov 02 22:01:21.565146 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:21.565116 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:01:29.363104 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:29.363072 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm"] Nov 02 22:01:29.469328 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:29.469294 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv"] Nov 02 22:01:34.364441 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.364401 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" podUID="76a5bef8-78a8-4e8e-9112-282d394069b4" containerName="payload-processing" containerID="cri-o://4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a" gracePeriod=25 Nov 02 22:01:34.470925 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.470885 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" podUID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" containerName="payload-pre-processing" containerID="cri-o://179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440" gracePeriod=25 Nov 02 22:01:34.633273 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.633217 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 22:01:34.677113 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.677082 4510 generic.go:356] "Generic (PLEG): container finished" podID="76a5bef8-78a8-4e8e-9112-282d394069b4" containerID="4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a" exitCode=0 Nov 02 22:01:34.677235 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.677116 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" event={"ID":"76a5bef8-78a8-4e8e-9112-282d394069b4","Type":"ContainerDied","Data":"4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a"} Nov 02 22:01:34.677235 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.677123 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" Nov 02 22:01:34.677235 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.677158 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm" event={"ID":"76a5bef8-78a8-4e8e-9112-282d394069b4","Type":"ContainerDied","Data":"b687f284740336a9827fd8c07d9a55ea38a24952286f6aff8d089edee9564f38"} Nov 02 22:01:34.677235 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.677178 4510 scope.go:122] "RemoveContainer" containerID="4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a" Nov 02 22:01:34.687644 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.687622 4510 scope.go:122] "RemoveContainer" containerID="4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a" Nov 02 22:01:34.687923 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:01:34.687902 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a\": container with ID starting with 4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a not found: ID does not exist" containerID="4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a" Nov 02 22:01:34.688001 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.687933 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a"} err="failed to get container status \"4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a\": rpc error: code = NotFound desc = could not find container \"4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a\": container with ID starting with 4674c94fd40842fcb002aebd181d7e99040a9d47edbbefbdd8079799d112553a not found: ID does not exist" Nov 02 22:01:34.777554 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.777508 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a5bef8-78a8-4e8e-9112-282d394069b4-kube-api-access-5rbph\" (UniqueName: \"kubernetes.io/projected/76a5bef8-78a8-4e8e-9112-282d394069b4-kube-api-access-5rbph\") pod \"76a5bef8-78a8-4e8e-9112-282d394069b4\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " Nov 02 22:01:34.777738 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.777650 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a5bef8-78a8-4e8e-9112-282d394069b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a5bef8-78a8-4e8e-9112-282d394069b4-plugins-config-volume\") pod \"76a5bef8-78a8-4e8e-9112-282d394069b4\" (UID: \"76a5bef8-78a8-4e8e-9112-282d394069b4\") " Nov 02 22:01:34.778153 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.778115 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a5bef8-78a8-4e8e-9112-282d394069b4-plugins-config-volume" pod "76a5bef8-78a8-4e8e-9112-282d394069b4" (UID: "76a5bef8-78a8-4e8e-9112-282d394069b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:34.778288 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.778238 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a5bef8-78a8-4e8e-9112-282d394069b4-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:34.779835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.779810 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a5bef8-78a8-4e8e-9112-282d394069b4-kube-api-access-5rbph" pod "76a5bef8-78a8-4e8e-9112-282d394069b4" (UID: "76a5bef8-78a8-4e8e-9112-282d394069b4"). InnerVolumeSpecName "kube-api-access-5rbph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:34.879708 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:34.879675 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rbph\" (UniqueName: \"kubernetes.io/projected/76a5bef8-78a8-4e8e-9112-282d394069b4-kube-api-access-5rbph\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:35.090338 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:35.090304 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm"] Nov 02 22:01:35.095193 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:35.095166 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-686bbf-7f776f59c7-ssgkm"] Nov 02 22:01:36.155019 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:36.154942 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a5bef8-78a8-4e8e-9112-282d394069b4" path="/var/lib/kubelet/pods/76a5bef8-78a8-4e8e-9112-282d394069b4/volumes" Nov 02 22:01:42.747345 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:42.747301 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" podUID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:01:45.534694 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:01:45.534591 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:01:52.748027 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:52.747985 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" podUID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:01:54.739505 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.739427 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 22:01:54.806788 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.806760 4510 generic.go:356] "Generic (PLEG): container finished" podID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" containerID="179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440" exitCode=0 Nov 02 22:01:54.806933 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.806801 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" Nov 02 22:01:54.806933 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.806838 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" event={"ID":"c59f0cae-4ae9-4841-8bca-b9d38ca799bc","Type":"ContainerDied","Data":"179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440"} Nov 02 22:01:54.806933 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.806885 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv" event={"ID":"c59f0cae-4ae9-4841-8bca-b9d38ca799bc","Type":"ContainerDied","Data":"eaaaadce2099ad196451b773f5adc6455669bc228439eb221803b215cee1fe1e"} Nov 02 22:01:54.806933 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.806906 4510 scope.go:122] "RemoveContainer" containerID="179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440" Nov 02 22:01:54.809315 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.809292 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-plugins-config-volume\") pod \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " Nov 02 22:01:54.809426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.809378 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-kube-api-access-sl4gl\" (UniqueName: \"kubernetes.io/projected/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-kube-api-access-sl4gl\") pod \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\" (UID: \"c59f0cae-4ae9-4841-8bca-b9d38ca799bc\") " Nov 02 22:01:54.809701 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.809679 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-plugins-config-volume" pod "c59f0cae-4ae9-4841-8bca-b9d38ca799bc" (UID: "c59f0cae-4ae9-4841-8bca-b9d38ca799bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:01:54.809929 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.809914 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:54.811343 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.811320 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-kube-api-access-sl4gl" pod "c59f0cae-4ae9-4841-8bca-b9d38ca799bc" (UID: "c59f0cae-4ae9-4841-8bca-b9d38ca799bc"). InnerVolumeSpecName "kube-api-access-sl4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:01:54.825453 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.821462 4510 scope.go:122] "RemoveContainer" containerID="179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440" Nov 02 22:01:54.825453 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:01:54.822037 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440\": container with ID starting with 179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440 not found: ID does not exist" containerID="179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440" Nov 02 22:01:54.825453 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.822065 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440"} err="failed to get container status \"179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440\": rpc error: code = NotFound desc = could not find container \"179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440\": container with ID starting with 179329ab10c3ff57190c07f550de1fadefa29a6f05796c5a6df6ee012731f440 not found: ID does not exist" Nov 02 22:01:54.911292 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:54.911265 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl4gl\" (UniqueName: \"kubernetes.io/projected/c59f0cae-4ae9-4841-8bca-b9d38ca799bc-kube-api-access-sl4gl\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:01:55.218678 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:55.218648 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv"] Nov 02 22:01:55.264996 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:55.264959 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-686bbf-56f69586dc-fczlv"] Nov 02 22:01:56.154851 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:01:56.154816 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" path="/var/lib/kubelet/pods/c59f0cae-4ae9-4841-8bca-b9d38ca799bc/volumes" Nov 02 22:02:01.997106 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:01.997071 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:02:02.702814 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:02.702780 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7"] Nov 02 22:02:02.703045 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:02.703010 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" podUID="a5818bc3-4b3b-4012-8587-ac0c506eb11a" containerName="maas-api" containerID="cri-o://d2354c14400a68860201f3b8a1f8ecc2302266c3552f83befac2b444c0d589c2" gracePeriod=30 Nov 02 22:02:02.861589 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:02.861551 4510 generic.go:356] "Generic (PLEG): container finished" podID="a5818bc3-4b3b-4012-8587-ac0c506eb11a" containerID="d2354c14400a68860201f3b8a1f8ecc2302266c3552f83befac2b444c0d589c2" exitCode=0 Nov 02 22:02:02.861909 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:02.861591 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" event={"ID":"a5818bc3-4b3b-4012-8587-ac0c506eb11a","Type":"ContainerDied","Data":"d2354c14400a68860201f3b8a1f8ecc2302266c3552f83befac2b444c0d589c2"} Nov 02 22:02:02.982246 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:02.982219 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 22:02:03.107929 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.107888 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-metrics-certs\") pod \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " Nov 02 22:02:03.108416 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.107943 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5818bc3-4b3b-4012-8587-ac0c506eb11a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5818bc3-4b3b-4012-8587-ac0c506eb11a-openshift-service-ca\") pod \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " Nov 02 22:02:03.108416 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.108041 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls\") pod \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " Nov 02 22:02:03.108416 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.108133 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5818bc3-4b3b-4012-8587-ac0c506eb11a-kube-api-access-x4lv6\" (UniqueName: \"kubernetes.io/projected/a5818bc3-4b3b-4012-8587-ac0c506eb11a-kube-api-access-x4lv6\") pod \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\" (UID: \"a5818bc3-4b3b-4012-8587-ac0c506eb11a\") " Nov 02 22:02:03.108658 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.108632 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5818bc3-4b3b-4012-8587-ac0c506eb11a-openshift-service-ca" pod "a5818bc3-4b3b-4012-8587-ac0c506eb11a" (UID: "a5818bc3-4b3b-4012-8587-ac0c506eb11a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:03.108892 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.108872 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5818bc3-4b3b-4012-8587-ac0c506eb11a-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:03.111103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.111067 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-metrics-certs" pod "a5818bc3-4b3b-4012-8587-ac0c506eb11a" (UID: "a5818bc3-4b3b-4012-8587-ac0c506eb11a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:03.112202 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.112173 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5818bc3-4b3b-4012-8587-ac0c506eb11a-kube-api-access-x4lv6" pod "a5818bc3-4b3b-4012-8587-ac0c506eb11a" (UID: "a5818bc3-4b3b-4012-8587-ac0c506eb11a"). InnerVolumeSpecName "kube-api-access-x4lv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:03.116657 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.116629 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls" pod "a5818bc3-4b3b-4012-8587-ac0c506eb11a" (UID: "a5818bc3-4b3b-4012-8587-ac0c506eb11a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:03.210244 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.210168 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:03.210244 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.210198 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4lv6\" (UniqueName: \"kubernetes.io/projected/a5818bc3-4b3b-4012-8587-ac0c506eb11a-kube-api-access-x4lv6\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:03.210244 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.210208 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5818bc3-4b3b-4012-8587-ac0c506eb11a-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:03.868644 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.868604 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" event={"ID":"a5818bc3-4b3b-4012-8587-ac0c506eb11a","Type":"ContainerDied","Data":"79bfb41166cf8298f29782ee1233d3e70a95263a90a2754870ab22a53e147b7a"} Nov 02 22:02:03.868828 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.868662 4510 scope.go:122] "RemoveContainer" containerID="d2354c14400a68860201f3b8a1f8ecc2302266c3552f83befac2b444c0d589c2" Nov 02 22:02:03.868828 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:03.868615 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7" Nov 02 22:02:04.020781 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:04.020750 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7"] Nov 02 22:02:04.066871 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:04.066836 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d4f799-8676d97b74-dt6b7"] Nov 02 22:02:04.155222 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:04.155128 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5818bc3-4b3b-4012-8587-ac0c506eb11a" path="/var/lib/kubelet/pods/a5818bc3-4b3b-4012-8587-ac0c506eb11a/volumes" Nov 02 22:02:08.504092 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.504054 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892"] Nov 02 22:02:08.504617 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.504349 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" podUID="abbc8325-78a6-4d47-ad11-b255b5428937" containerName="maas-api" containerID="cri-o://2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4" gracePeriod=30 Nov 02 22:02:08.778922 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.778894 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 22:02:08.877663 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.877630 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-metrics-certs\") pod \"abbc8325-78a6-4d47-ad11-b255b5428937\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " Nov 02 22:02:08.877838 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.877714 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbc8325-78a6-4d47-ad11-b255b5428937-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbc8325-78a6-4d47-ad11-b255b5428937-openshift-service-ca\") pod \"abbc8325-78a6-4d47-ad11-b255b5428937\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " Nov 02 22:02:08.878662 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.877889 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbc8325-78a6-4d47-ad11-b255b5428937-kube-api-access-dnzbp\" (UniqueName: \"kubernetes.io/projected/abbc8325-78a6-4d47-ad11-b255b5428937-kube-api-access-dnzbp\") pod \"abbc8325-78a6-4d47-ad11-b255b5428937\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " Nov 02 22:02:08.878662 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.878031 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls\") pod \"abbc8325-78a6-4d47-ad11-b255b5428937\" (UID: \"abbc8325-78a6-4d47-ad11-b255b5428937\") " Nov 02 22:02:08.878662 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.878091 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbc8325-78a6-4d47-ad11-b255b5428937-openshift-service-ca" pod "abbc8325-78a6-4d47-ad11-b255b5428937" (UID: "abbc8325-78a6-4d47-ad11-b255b5428937"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:08.878662 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.878632 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbc8325-78a6-4d47-ad11-b255b5428937-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:08.880493 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.880462 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbc8325-78a6-4d47-ad11-b255b5428937-kube-api-access-dnzbp" pod "abbc8325-78a6-4d47-ad11-b255b5428937" (UID: "abbc8325-78a6-4d47-ad11-b255b5428937"). InnerVolumeSpecName "kube-api-access-dnzbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:08.880724 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.880514 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-metrics-certs" pod "abbc8325-78a6-4d47-ad11-b255b5428937" (UID: "abbc8325-78a6-4d47-ad11-b255b5428937"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:08.880724 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.880626 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls" pod "abbc8325-78a6-4d47-ad11-b255b5428937" (UID: "abbc8325-78a6-4d47-ad11-b255b5428937"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:08.905898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.905585 4510 generic.go:356] "Generic (PLEG): container finished" podID="abbc8325-78a6-4d47-ad11-b255b5428937" containerID="2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4" exitCode=0 Nov 02 22:02:08.905898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.905650 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" Nov 02 22:02:08.905898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.905655 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" event={"ID":"abbc8325-78a6-4d47-ad11-b255b5428937","Type":"ContainerDied","Data":"2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4"} Nov 02 22:02:08.905898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.905731 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892" event={"ID":"abbc8325-78a6-4d47-ad11-b255b5428937","Type":"ContainerDied","Data":"0ba225c3ce27415eb7c0c52806b184cc2c87bc5acf99ff4797a2f36b211049bd"} Nov 02 22:02:08.905898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.905768 4510 scope.go:122] "RemoveContainer" containerID="2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4" Nov 02 22:02:08.918331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.918313 4510 scope.go:122] "RemoveContainer" containerID="2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4" Nov 02 22:02:08.918866 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:02:08.918836 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4\": container with ID starting with 2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4 not found: ID does not exist" containerID="2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4" Nov 02 22:02:08.919045 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.919012 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4"} err="failed to get container status \"2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4\": rpc error: code = NotFound desc = could not find container \"2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4\": container with ID starting with 2d129ec2d0b77ff4a425469c3b96169b2ddddc40f251d6ba194551057a136ce4 not found: ID does not exist" Nov 02 22:02:08.979951 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.979925 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnzbp\" (UniqueName: \"kubernetes.io/projected/abbc8325-78a6-4d47-ad11-b255b5428937-kube-api-access-dnzbp\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:08.979951 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.979947 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:08.980120 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:08.979958 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abbc8325-78a6-4d47-ad11-b255b5428937-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:09.058417 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:09.058340 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892"] Nov 02 22:02:09.062211 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:09.062184 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-686bbf-6589b9d846-29892"] Nov 02 22:02:10.155682 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:10.155648 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbc8325-78a6-4d47-ad11-b255b5428937" path="/var/lib/kubelet/pods/abbc8325-78a6-4d47-ad11-b255b5428937/volumes" Nov 02 22:02:15.530809 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:02:15.530772 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:02:17.364369 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:17.364335 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd"] Nov 02 22:02:17.364813 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:17.364587 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" containerID="cri-o://b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a" gracePeriod=30 Nov 02 22:02:22.639153 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.639057 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 22:02:22.744024 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.743988 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-credential-socket\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744190 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744030 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467f2a6a-2feb-43ab-b240-d909953c0cb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f2a6a-2feb-43ab-b240-d909953c0cb5-istiod-ca-cert\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744190 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744050 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-data\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744190 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744075 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-certs\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744190 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744102 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-envoy\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744190 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744165 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-socket\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744228 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-kube-api-access-fxn4z\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-kube-api-access-fxn4z\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744296 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-token\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-token\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744322 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-credential-socket" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:22.744472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744369 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-podinfo\") pod \"467f2a6a-2feb-43ab-b240-d909953c0cb5\" (UID: \"467f2a6a-2feb-43ab-b240-d909953c0cb5\") " Nov 02 22:02:22.744728 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744476 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-socket" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:22.744728 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744500 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467f2a6a-2feb-43ab-b240-d909953c0cb5-istiod-ca-cert" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:22.744728 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744665 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-data" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:22.744894 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.744771 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-certs" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:22.745024 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.745003 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.745118 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.745029 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/467f2a6a-2feb-43ab-b240-d909953c0cb5-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.745118 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.745044 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.745118 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.745057 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.745118 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.745074 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.746664 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.746634 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-podinfo" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:02:22.746664 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.746644 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-envoy" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:22.746835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.746717 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-kube-api-access-fxn4z" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "kube-api-access-fxn4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:22.746835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.746731 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-token" pod "467f2a6a-2feb-43ab-b240-d909953c0cb5" (UID: "467f2a6a-2feb-43ab-b240-d909953c0cb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:22.846850 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.846819 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxn4z\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-kube-api-access-fxn4z\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.846850 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.846849 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.847049 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.846859 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.847049 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.846868 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/467f2a6a-2feb-43ab-b240-d909953c0cb5-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:22.991450 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.991366 4510 generic.go:356] "Generic (PLEG): container finished" podID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerID="b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a" exitCode=0 Nov 02 22:02:22.991450 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.991426 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" Nov 02 22:02:22.991647 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.991458 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" event={"ID":"467f2a6a-2feb-43ab-b240-d909953c0cb5","Type":"ContainerDied","Data":"b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a"} Nov 02 22:02:22.991647 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.991516 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd" event={"ID":"467f2a6a-2feb-43ab-b240-d909953c0cb5","Type":"ContainerDied","Data":"b859f8d303da6af3fca281f3691ed0c7a2edf9ede380f8dfdda26a3af0e17a23"} Nov 02 22:02:22.991647 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:22.991545 4510 scope.go:122] "RemoveContainer" containerID="b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a" Nov 02 22:02:23.002856 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:23.002839 4510 scope.go:122] "RemoveContainer" containerID="ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506" Nov 02 22:02:23.011753 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:23.011737 4510 scope.go:122] "RemoveContainer" containerID="b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a" Nov 02 22:02:23.012006 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:02:23.011987 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a\": container with ID starting with b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a not found: ID does not exist" containerID="b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a" Nov 02 22:02:23.012057 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:23.012015 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a"} err="failed to get container status \"b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a\": rpc error: code = NotFound desc = could not find container \"b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a\": container with ID starting with b25aa5e9562c0087daf34dad311950ad966eba0745008c38661e71de899ae23a not found: ID does not exist" Nov 02 22:02:23.012057 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:23.012032 4510 scope.go:122] "RemoveContainer" containerID="ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506" Nov 02 22:02:23.012235 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:02:23.012219 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506\": container with ID starting with ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506 not found: ID does not exist" containerID="ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506" Nov 02 22:02:23.012276 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:23.012240 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506"} err="failed to get container status \"ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506\": rpc error: code = NotFound desc = could not find container \"ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506\": container with ID starting with ce1447f7de532feec3043233b5b805d26d76a253d47781367cd1e5a87e432506 not found: ID does not exist" Nov 02 22:02:23.145943 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:23.145913 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd"] Nov 02 22:02:23.150382 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:23.150357 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-686bbf-openshift-default-65b6c69fbc-vwvdd"] Nov 02 22:02:24.185460 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:24.185414 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" path="/var/lib/kubelet/pods/467f2a6a-2feb-43ab-b240-d909953c0cb5/volumes" Nov 02 22:02:29.441346 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.441312 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct"] Nov 02 22:02:29.441799 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.441551 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" podUID="71a6aa31-bb91-41cd-a91b-d83d613e2275" containerName="maas-api" containerID="cri-o://8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd" gracePeriod=30 Nov 02 22:02:29.562912 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.562881 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" podUID="71a6aa31-bb91-41cd-a91b-d83d613e2275" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.135:8443/health\": dial tcp 10.134.0.135:8443: connect: connection refused" Nov 02 22:02:29.707252 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.707182 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 22:02:29.845194 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.845154 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a6aa31-bb91-41cd-a91b-d83d613e2275-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6aa31-bb91-41cd-a91b-d83d613e2275-openshift-service-ca\") pod \"71a6aa31-bb91-41cd-a91b-d83d613e2275\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " Nov 02 22:02:29.845365 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.845229 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-metrics-certs\") pod \"71a6aa31-bb91-41cd-a91b-d83d613e2275\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " Nov 02 22:02:29.845481 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.845464 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls\") pod \"71a6aa31-bb91-41cd-a91b-d83d613e2275\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " Nov 02 22:02:29.845565 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.845530 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a6aa31-bb91-41cd-a91b-d83d613e2275-kube-api-access-vg4d6\" (UniqueName: \"kubernetes.io/projected/71a6aa31-bb91-41cd-a91b-d83d613e2275-kube-api-access-vg4d6\") pod \"71a6aa31-bb91-41cd-a91b-d83d613e2275\" (UID: \"71a6aa31-bb91-41cd-a91b-d83d613e2275\") " Nov 02 22:02:29.845637 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.845607 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a6aa31-bb91-41cd-a91b-d83d613e2275-openshift-service-ca" pod "71a6aa31-bb91-41cd-a91b-d83d613e2275" (UID: "71a6aa31-bb91-41cd-a91b-d83d613e2275"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:02:29.846120 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.846098 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a6aa31-bb91-41cd-a91b-d83d613e2275-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:29.848010 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.847983 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls" pod "71a6aa31-bb91-41cd-a91b-d83d613e2275" (UID: "71a6aa31-bb91-41cd-a91b-d83d613e2275"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:29.848010 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.847993 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-metrics-certs" pod "71a6aa31-bb91-41cd-a91b-d83d613e2275" (UID: "71a6aa31-bb91-41cd-a91b-d83d613e2275"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:29.848222 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.848208 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a6aa31-bb91-41cd-a91b-d83d613e2275-kube-api-access-vg4d6" pod "71a6aa31-bb91-41cd-a91b-d83d613e2275" (UID: "71a6aa31-bb91-41cd-a91b-d83d613e2275"). InnerVolumeSpecName "kube-api-access-vg4d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:29.947978 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.947943 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:29.947978 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.947979 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4d6\" (UniqueName: \"kubernetes.io/projected/71a6aa31-bb91-41cd-a91b-d83d613e2275-kube-api-access-vg4d6\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:29.948214 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:29.947995 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71a6aa31-bb91-41cd-a91b-d83d613e2275-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:30.033631 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.033543 4510 generic.go:356] "Generic (PLEG): container finished" podID="71a6aa31-bb91-41cd-a91b-d83d613e2275" containerID="8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd" exitCode=0 Nov 02 22:02:30.033631 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.033582 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" Nov 02 22:02:30.033828 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.033551 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" event={"ID":"71a6aa31-bb91-41cd-a91b-d83d613e2275","Type":"ContainerDied","Data":"8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd"} Nov 02 22:02:30.033828 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.033686 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct" event={"ID":"71a6aa31-bb91-41cd-a91b-d83d613e2275","Type":"ContainerDied","Data":"5e4a7ba40ee614d96442e4fd5542c25e491b00ddce0f26c55ce8cf685a5ea0a1"} Nov 02 22:02:30.033828 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.033709 4510 scope.go:122] "RemoveContainer" containerID="8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd" Nov 02 22:02:30.051684 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.051660 4510 scope.go:122] "RemoveContainer" containerID="8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd" Nov 02 22:02:30.051990 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:02:30.051966 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd\": container with ID starting with 8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd not found: ID does not exist" containerID="8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd" Nov 02 22:02:30.052052 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.051997 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd"} err="failed to get container status \"8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd\": rpc error: code = NotFound desc = could not find container \"8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd\": container with ID starting with 8ad4d4f40418a7f51757db7db53c212ad6985edbde27485ea776afa7d92a35fd not found: ID does not exist" Nov 02 22:02:30.187122 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.187083 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct"] Nov 02 22:02:30.191952 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:30.191915 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-11537e-7bf4d7569c-pwzct"] Nov 02 22:02:32.155653 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:32.155614 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a6aa31-bb91-41cd-a91b-d83d613e2275" path="/var/lib/kubelet/pods/71a6aa31-bb91-41cd-a91b-d83d613e2275/volumes" Nov 02 22:02:34.636689 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:34.636653 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc"] Nov 02 22:02:45.534969 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:02:45.534934 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:02:47.287161 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:47.287129 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv"] Nov 02 22:02:47.332838 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:47.332806 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd"] Nov 02 22:02:49.657220 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.657157 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" podUID="b8b15b7e-e88e-47a2-9472-a26694c11ce9" containerName="main" containerID="cri-o://360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb" gracePeriod=45 Nov 02 22:02:49.915314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.915258 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 22:02:49.996148 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996111 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-model-cache\") pod \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " Nov 02 22:02:49.996332 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996180 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-dshm\") pod \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " Nov 02 22:02:49.996332 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996235 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b15b7e-e88e-47a2-9472-a26694c11ce9-kube-api-access-dknd9\" (UniqueName: \"kubernetes.io/projected/b8b15b7e-e88e-47a2-9472-a26694c11ce9-kube-api-access-dknd9\") pod \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " Nov 02 22:02:49.996332 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996295 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tls-certs\") pod \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " Nov 02 22:02:49.996332 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996327 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tmp-dir\") pod \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " Nov 02 22:02:49.996589 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996438 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-model-cache" pod "b8b15b7e-e88e-47a2-9472-a26694c11ce9" (UID: "b8b15b7e-e88e-47a2-9472-a26694c11ce9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:49.996589 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996452 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-home\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-home\") pod \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\" (UID: \"b8b15b7e-e88e-47a2-9472-a26694c11ce9\") " Nov 02 22:02:49.996728 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996691 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tmp-dir" pod "b8b15b7e-e88e-47a2-9472-a26694c11ce9" (UID: "b8b15b7e-e88e-47a2-9472-a26694c11ce9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:49.996728 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.996713 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-home" pod "b8b15b7e-e88e-47a2-9472-a26694c11ce9" (UID: "b8b15b7e-e88e-47a2-9472-a26694c11ce9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:49.997038 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.997012 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:49.997152 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.997041 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:49.997152 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.997055 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:49.998566 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.998539 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-dshm" pod "b8b15b7e-e88e-47a2-9472-a26694c11ce9" (UID: "b8b15b7e-e88e-47a2-9472-a26694c11ce9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:02:49.998660 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.998633 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tls-certs" pod "b8b15b7e-e88e-47a2-9472-a26694c11ce9" (UID: "b8b15b7e-e88e-47a2-9472-a26694c11ce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:02:49.998704 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:49.998658 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b15b7e-e88e-47a2-9472-a26694c11ce9-kube-api-access-dknd9" pod "b8b15b7e-e88e-47a2-9472-a26694c11ce9" (UID: "b8b15b7e-e88e-47a2-9472-a26694c11ce9"). InnerVolumeSpecName "kube-api-access-dknd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:02:50.098392 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.098360 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8b15b7e-e88e-47a2-9472-a26694c11ce9-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.098392 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.098388 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dknd9\" (UniqueName: \"kubernetes.io/projected/b8b15b7e-e88e-47a2-9472-a26694c11ce9-kube-api-access-dknd9\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.098392 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.098398 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8b15b7e-e88e-47a2-9472-a26694c11ce9-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:02:50.156755 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.156714 4510 generic.go:356] "Generic (PLEG): container finished" podID="b8b15b7e-e88e-47a2-9472-a26694c11ce9" containerID="360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb" exitCode=0 Nov 02 22:02:50.156938 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.156759 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" Nov 02 22:02:50.156938 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.156797 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" event={"ID":"b8b15b7e-e88e-47a2-9472-a26694c11ce9","Type":"ContainerDied","Data":"360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb"} Nov 02 22:02:50.156938 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.156844 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc" event={"ID":"b8b15b7e-e88e-47a2-9472-a26694c11ce9","Type":"ContainerDied","Data":"adcdc51f0cf0cd6b84b2abea67b568a5229a7df3c34defe39bedf89da2709c75"} Nov 02 22:02:50.156938 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.156871 4510 scope.go:122] "RemoveContainer" containerID="360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb" Nov 02 22:02:50.168380 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.168362 4510 scope.go:122] "RemoveContainer" containerID="360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb" Nov 02 22:02:50.168680 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:02:50.168661 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb\": container with ID starting with 360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb not found: ID does not exist" containerID="360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb" Nov 02 22:02:50.168737 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.168688 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb"} err="failed to get container status \"360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb\": rpc error: code = NotFound desc = could not find container \"360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb\": container with ID starting with 360e94cfad5279f534bcc8645831064aa2b171c3532195f8abbd817230746dfb not found: ID does not exist" Nov 02 22:02:50.308417 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.308387 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc"] Nov 02 22:02:50.311294 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:50.311268 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3566f9/test-model-3566f9-kserve-55486df8c7-t7qpc"] Nov 02 22:02:52.154422 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:52.154386 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b15b7e-e88e-47a2-9472-a26694c11ce9" path="/var/lib/kubelet/pods/b8b15b7e-e88e-47a2-9472-a26694c11ce9/volumes" Nov 02 22:02:53.296063 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:53.296032 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv"] Nov 02 22:02:58.296737 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:58.296681 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" podUID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" containerName="payload-pre-processing" containerID="cri-o://9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b" gracePeriod=25 Nov 02 22:02:59.375039 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:59.375003 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" podUID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:02:59.834247 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:02:59.834166 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:03:02.309735 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.309689 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" containerID="cri-o://60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab" gracePeriod=45 Nov 02 22:03:02.324422 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.324385 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Nov 02 22:03:02.356745 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.356707 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" containerID="cri-o://f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e" gracePeriod=45 Nov 02 22:03:02.611957 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.611924 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 22:03:02.639622 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.639594 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 22:03:02.733398 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733369 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe4486fa-40c4-4a43-90b4-07e436859993-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4486fa-40c4-4a43-90b4-07e436859993-tls-certs\") pod \"fe4486fa-40c4-4a43-90b4-07e436859993\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " Nov 02 22:03:02.733576 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733435 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a38d552-966f-4ca0-8f60-be2db4e772fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a38d552-966f-4ca0-8f60-be2db4e772fe-tls-certs\") pod \"2a38d552-966f-4ca0-8f60-be2db4e772fe\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " Nov 02 22:03:02.733576 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733472 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-tmp-dir\") pod \"2a38d552-966f-4ca0-8f60-be2db4e772fe\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " Nov 02 22:03:02.733576 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733531 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-model-cache\") pod \"fe4486fa-40c4-4a43-90b4-07e436859993\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " Nov 02 22:03:02.733769 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733581 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-home\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-home\") pod \"2a38d552-966f-4ca0-8f60-be2db4e772fe\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " Nov 02 22:03:02.733769 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733619 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-tmp-dir\") pod \"fe4486fa-40c4-4a43-90b4-07e436859993\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " Nov 02 22:03:02.733769 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733680 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-home\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-home\") pod \"fe4486fa-40c4-4a43-90b4-07e436859993\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " Nov 02 22:03:02.733942 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733780 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4486fa-40c4-4a43-90b4-07e436859993-kube-api-access-9px6b\" (UniqueName: \"kubernetes.io/projected/fe4486fa-40c4-4a43-90b4-07e436859993-kube-api-access-9px6b\") pod \"fe4486fa-40c4-4a43-90b4-07e436859993\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " Nov 02 22:03:02.733942 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733795 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-model-cache" pod "fe4486fa-40c4-4a43-90b4-07e436859993" (UID: "fe4486fa-40c4-4a43-90b4-07e436859993"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.733942 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733863 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-dshm\") pod \"2a38d552-966f-4ca0-8f60-be2db4e772fe\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " Nov 02 22:03:02.733942 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733928 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a38d552-966f-4ca0-8f60-be2db4e772fe-kube-api-access-v77hg\" (UniqueName: \"kubernetes.io/projected/2a38d552-966f-4ca0-8f60-be2db4e772fe-kube-api-access-v77hg\") pod \"2a38d552-966f-4ca0-8f60-be2db4e772fe\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " Nov 02 22:03:02.734225 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.733961 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-model-cache\") pod \"2a38d552-966f-4ca0-8f60-be2db4e772fe\" (UID: \"2a38d552-966f-4ca0-8f60-be2db4e772fe\") " Nov 02 22:03:02.734225 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734029 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-dshm\") pod \"fe4486fa-40c4-4a43-90b4-07e436859993\" (UID: \"fe4486fa-40c4-4a43-90b4-07e436859993\") " Nov 02 22:03:02.734331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734252 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-tmp-dir" pod "fe4486fa-40c4-4a43-90b4-07e436859993" (UID: "fe4486fa-40c4-4a43-90b4-07e436859993"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.734988 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734477 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-home" pod "2a38d552-966f-4ca0-8f60-be2db4e772fe" (UID: "2a38d552-966f-4ca0-8f60-be2db4e772fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.734988 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734631 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.734988 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734652 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.734988 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734718 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.734988 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734778 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-tmp-dir" pod "2a38d552-966f-4ca0-8f60-be2db4e772fe" (UID: "2a38d552-966f-4ca0-8f60-be2db4e772fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.734988 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734911 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-home" pod "fe4486fa-40c4-4a43-90b4-07e436859993" (UID: "fe4486fa-40c4-4a43-90b4-07e436859993"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.735396 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.734991 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-model-cache" pod "2a38d552-966f-4ca0-8f60-be2db4e772fe" (UID: "2a38d552-966f-4ca0-8f60-be2db4e772fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.736406 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.736374 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4486fa-40c4-4a43-90b4-07e436859993-tls-certs" pod "fe4486fa-40c4-4a43-90b4-07e436859993" (UID: "fe4486fa-40c4-4a43-90b4-07e436859993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:02.736406 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.736395 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4486fa-40c4-4a43-90b4-07e436859993-kube-api-access-9px6b" pod "fe4486fa-40c4-4a43-90b4-07e436859993" (UID: "fe4486fa-40c4-4a43-90b4-07e436859993"). InnerVolumeSpecName "kube-api-access-9px6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:02.736630 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.736484 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a38d552-966f-4ca0-8f60-be2db4e772fe-tls-certs" pod "2a38d552-966f-4ca0-8f60-be2db4e772fe" (UID: "2a38d552-966f-4ca0-8f60-be2db4e772fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:02.736931 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.736903 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-dshm" pod "2a38d552-966f-4ca0-8f60-be2db4e772fe" (UID: "2a38d552-966f-4ca0-8f60-be2db4e772fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.737029 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.737008 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-dshm" pod "fe4486fa-40c4-4a43-90b4-07e436859993" (UID: "fe4486fa-40c4-4a43-90b4-07e436859993"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:03:02.737564 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.737509 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a38d552-966f-4ca0-8f60-be2db4e772fe-kube-api-access-v77hg" pod "2a38d552-966f-4ca0-8f60-be2db4e772fe" (UID: "2a38d552-966f-4ca0-8f60-be2db4e772fe"). InnerVolumeSpecName "kube-api-access-v77hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:02.836109 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836020 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a38d552-966f-4ca0-8f60-be2db4e772fe-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836109 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836061 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836109 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836075 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836109 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836090 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9px6b\" (UniqueName: \"kubernetes.io/projected/fe4486fa-40c4-4a43-90b4-07e436859993-kube-api-access-9px6b\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836109 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836108 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836379 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836123 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77hg\" (UniqueName: \"kubernetes.io/projected/2a38d552-966f-4ca0-8f60-be2db4e772fe-kube-api-access-v77hg\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836379 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836138 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a38d552-966f-4ca0-8f60-be2db4e772fe-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836379 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836154 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe4486fa-40c4-4a43-90b4-07e436859993-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:02.836379 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:02.836170 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe4486fa-40c4-4a43-90b4-07e436859993-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:03.234323 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.234291 4510 generic.go:356] "Generic (PLEG): container finished" podID="fe4486fa-40c4-4a43-90b4-07e436859993" containerID="f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e" exitCode=0 Nov 02 22:03:03.234475 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.234338 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" Nov 02 22:03:03.234475 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.234373 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" event={"ID":"fe4486fa-40c4-4a43-90b4-07e436859993","Type":"ContainerDied","Data":"f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e"} Nov 02 22:03:03.234475 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.234417 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd" event={"ID":"fe4486fa-40c4-4a43-90b4-07e436859993","Type":"ContainerDied","Data":"c9aad43002306cfe6e13d1a924a35db25ea5434fcc34395a45537a65d27468fa"} Nov 02 22:03:03.234475 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.234440 4510 scope.go:122] "RemoveContainer" containerID="f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e" Nov 02 22:03:03.235891 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.235865 4510 generic.go:356] "Generic (PLEG): container finished" podID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerID="60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab" exitCode=0 Nov 02 22:03:03.236006 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.235907 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" Nov 02 22:03:03.236006 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.235941 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" event={"ID":"2a38d552-966f-4ca0-8f60-be2db4e772fe","Type":"ContainerDied","Data":"60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab"} Nov 02 22:03:03.236006 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.235973 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv" event={"ID":"2a38d552-966f-4ca0-8f60-be2db4e772fe","Type":"ContainerDied","Data":"c109385a461c5a0c4df74329d94a2b8a5395ebfc0340e8c4bdffc9adcd86a4cd"} Nov 02 22:03:03.244477 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.244462 4510 scope.go:122] "RemoveContainer" containerID="f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e" Nov 02 22:03:03.244757 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:03.244741 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e\": container with ID starting with f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e not found: ID does not exist" containerID="f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e" Nov 02 22:03:03.244823 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.244765 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e"} err="failed to get container status \"f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e\": rpc error: code = NotFound desc = could not find container \"f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e\": container with ID starting with f125222c5c44c1275c26b7fdd63a2d0e170b5132169a2143f5481277b2c2a98e not found: ID does not exist" Nov 02 22:03:03.244823 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.244782 4510 scope.go:122] "RemoveContainer" containerID="60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab" Nov 02 22:03:03.253242 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.253225 4510 scope.go:122] "RemoveContainer" containerID="60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab" Nov 02 22:03:03.253493 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:03.253476 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab\": container with ID starting with 60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab not found: ID does not exist" containerID="60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab" Nov 02 22:03:03.253547 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.253500 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab"} err="failed to get container status \"60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab\": rpc error: code = NotFound desc = could not find container \"60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab\": container with ID starting with 60b32de1853e4fce484faef6af2f650bc162ca7c1a45a95d20adc419304d19ab not found: ID does not exist" Nov 02 22:03:03.385830 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.385797 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd"] Nov 02 22:03:03.432001 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.431965 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-w1-c2dd3b-kserve-7bfbfccb48-gxjfd"] Nov 02 22:03:03.532133 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.532023 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv"] Nov 02 22:03:03.535549 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:03.535512 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c2dd3b/distinct-2-w1-c2dd3b-kserve-5b4d475fd7-zl4nv"] Nov 02 22:03:04.154637 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:04.154606 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" path="/var/lib/kubelet/pods/2a38d552-966f-4ca0-8f60-be2db4e772fe/volumes" Nov 02 22:03:04.155025 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:04.155012 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" path="/var/lib/kubelet/pods/fe4486fa-40c4-4a43-90b4-07e436859993/volumes" Nov 02 22:03:09.012371 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.012287 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j"] Nov 02 22:03:09.012839 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.012588 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" podUID="70ad2a87-ef27-4f57-990d-860fd992a0a3" containerName="maas-api" containerID="cri-o://52c34f1617a8c60ba7bbdb62c76bc91bb0e982dd8fd3ab58e204dbdede795cad" gracePeriod=30 Nov 02 22:03:09.273281 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.273218 4510 generic.go:356] "Generic (PLEG): container finished" podID="70ad2a87-ef27-4f57-990d-860fd992a0a3" containerID="52c34f1617a8c60ba7bbdb62c76bc91bb0e982dd8fd3ab58e204dbdede795cad" exitCode=0 Nov 02 22:03:09.273281 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.273253 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" event={"ID":"70ad2a87-ef27-4f57-990d-860fd992a0a3","Type":"ContainerDied","Data":"52c34f1617a8c60ba7bbdb62c76bc91bb0e982dd8fd3ab58e204dbdede795cad"} Nov 02 22:03:09.273431 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.273292 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" event={"ID":"70ad2a87-ef27-4f57-990d-860fd992a0a3","Type":"ContainerDied","Data":"c683c7a85a3455d34622c4bef35d63d95eb336571a005b267fee427dfff2d0a0"} Nov 02 22:03:09.273431 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.273308 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c683c7a85a3455d34622c4bef35d63d95eb336571a005b267fee427dfff2d0a0" Nov 02 22:03:09.278274 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.278252 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 22:03:09.375467 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.375436 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" podUID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:03:09.410226 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.410186 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-metrics-certs\") pod \"70ad2a87-ef27-4f57-990d-860fd992a0a3\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " Nov 02 22:03:09.410394 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.410298 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls\") pod \"70ad2a87-ef27-4f57-990d-860fd992a0a3\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " Nov 02 22:03:09.410460 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.410395 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ad2a87-ef27-4f57-990d-860fd992a0a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ad2a87-ef27-4f57-990d-860fd992a0a3-openshift-service-ca\") pod \"70ad2a87-ef27-4f57-990d-860fd992a0a3\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " Nov 02 22:03:09.410460 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.410435 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ad2a87-ef27-4f57-990d-860fd992a0a3-kube-api-access-jmmxf\" (UniqueName: \"kubernetes.io/projected/70ad2a87-ef27-4f57-990d-860fd992a0a3-kube-api-access-jmmxf\") pod \"70ad2a87-ef27-4f57-990d-860fd992a0a3\" (UID: \"70ad2a87-ef27-4f57-990d-860fd992a0a3\") " Nov 02 22:03:09.410840 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.410810 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ad2a87-ef27-4f57-990d-860fd992a0a3-openshift-service-ca" pod "70ad2a87-ef27-4f57-990d-860fd992a0a3" (UID: "70ad2a87-ef27-4f57-990d-860fd992a0a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:09.412419 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.412389 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-metrics-certs" pod "70ad2a87-ef27-4f57-990d-860fd992a0a3" (UID: "70ad2a87-ef27-4f57-990d-860fd992a0a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:09.412542 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.412472 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ad2a87-ef27-4f57-990d-860fd992a0a3-kube-api-access-jmmxf" pod "70ad2a87-ef27-4f57-990d-860fd992a0a3" (UID: "70ad2a87-ef27-4f57-990d-860fd992a0a3"). InnerVolumeSpecName "kube-api-access-jmmxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:09.412836 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.412813 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls" pod "70ad2a87-ef27-4f57-990d-860fd992a0a3" (UID: "70ad2a87-ef27-4f57-990d-860fd992a0a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:09.512806 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.512768 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:09.512806 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.512800 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70ad2a87-ef27-4f57-990d-860fd992a0a3-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:09.512806 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.512811 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70ad2a87-ef27-4f57-990d-860fd992a0a3-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:09.513047 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:09.512820 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmxf\" (UniqueName: \"kubernetes.io/projected/70ad2a87-ef27-4f57-990d-860fd992a0a3-kube-api-access-jmmxf\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:10.101617 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.101581 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw"] Nov 02 22:03:10.102056 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.101796 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" podUID="4233e721-fed9-4272-b083-700f52ffecda" containerName="maas-api" containerID="cri-o://1945dec15b030872ce7799a5d67f362c5cf560975181d5c94c152926e187025d" gracePeriod=30 Nov 02 22:03:10.283599 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.283558 4510 generic.go:356] "Generic (PLEG): container finished" podID="4233e721-fed9-4272-b083-700f52ffecda" containerID="1945dec15b030872ce7799a5d67f362c5cf560975181d5c94c152926e187025d" exitCode=0 Nov 02 22:03:10.283736 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.283628 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" event={"ID":"4233e721-fed9-4272-b083-700f52ffecda","Type":"ContainerDied","Data":"1945dec15b030872ce7799a5d67f362c5cf560975181d5c94c152926e187025d"} Nov 02 22:03:10.283992 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.283819 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j" Nov 02 22:03:10.356928 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.356871 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 22:03:10.423004 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.422965 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls\") pod \"4233e721-fed9-4272-b083-700f52ffecda\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " Nov 02 22:03:10.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.423064 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4233e721-fed9-4272-b083-700f52ffecda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4233e721-fed9-4272-b083-700f52ffecda-openshift-service-ca\") pod \"4233e721-fed9-4272-b083-700f52ffecda\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " Nov 02 22:03:10.423191 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.423159 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-metrics-certs\") pod \"4233e721-fed9-4272-b083-700f52ffecda\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " Nov 02 22:03:10.423297 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.423257 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4233e721-fed9-4272-b083-700f52ffecda-kube-api-access-bqnk9\" (UniqueName: \"kubernetes.io/projected/4233e721-fed9-4272-b083-700f52ffecda-kube-api-access-bqnk9\") pod \"4233e721-fed9-4272-b083-700f52ffecda\" (UID: \"4233e721-fed9-4272-b083-700f52ffecda\") " Nov 02 22:03:10.423455 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.423425 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4233e721-fed9-4272-b083-700f52ffecda-openshift-service-ca" pod "4233e721-fed9-4272-b083-700f52ffecda" (UID: "4233e721-fed9-4272-b083-700f52ffecda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:10.423795 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.423768 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4233e721-fed9-4272-b083-700f52ffecda-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:10.425203 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.425173 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls" pod "4233e721-fed9-4272-b083-700f52ffecda" (UID: "4233e721-fed9-4272-b083-700f52ffecda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:10.425286 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.425222 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-metrics-certs" pod "4233e721-fed9-4272-b083-700f52ffecda" (UID: "4233e721-fed9-4272-b083-700f52ffecda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:10.425342 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.425284 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4233e721-fed9-4272-b083-700f52ffecda-kube-api-access-bqnk9" pod "4233e721-fed9-4272-b083-700f52ffecda" (UID: "4233e721-fed9-4272-b083-700f52ffecda"). InnerVolumeSpecName "kube-api-access-bqnk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:10.436735 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.436713 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j"] Nov 02 22:03:10.446383 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.446359 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3566f9-5cc6c8dcc7-pz98j"] Nov 02 22:03:10.525162 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.525132 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqnk9\" (UniqueName: \"kubernetes.io/projected/4233e721-fed9-4272-b083-700f52ffecda-kube-api-access-bqnk9\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:10.525162 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.525160 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:10.525162 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:10.525171 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4233e721-fed9-4272-b083-700f52ffecda-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:11.290144 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:11.290116 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" Nov 02 22:03:11.290616 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:11.290144 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw" event={"ID":"4233e721-fed9-4272-b083-700f52ffecda","Type":"ContainerDied","Data":"cbbc622d967f2ef0bba19c5a6f2d259190f13bef9eac0d5b41c824a4e510a017"} Nov 02 22:03:11.290616 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:11.290196 4510 scope.go:122] "RemoveContainer" containerID="1945dec15b030872ce7799a5d67f362c5cf560975181d5c94c152926e187025d" Nov 02 22:03:11.442793 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:11.442758 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw"] Nov 02 22:03:11.489186 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:11.489158 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f4a242-8895ff589-t7cmw"] Nov 02 22:03:12.154719 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:12.154682 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4233e721-fed9-4272-b083-700f52ffecda" path="/var/lib/kubelet/pods/4233e721-fed9-4272-b083-700f52ffecda/volumes" Nov 02 22:03:12.155174 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:12.155158 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ad2a87-ef27-4f57-990d-860fd992a0a3" path="/var/lib/kubelet/pods/70ad2a87-ef27-4f57-990d-860fd992a0a3/volumes" Nov 02 22:03:15.530352 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:15.530314 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:03:18.560909 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:18.560855 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 22:03:18.714135 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:18.714105 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-kube-api-access-hgx62\" (UniqueName: \"kubernetes.io/projected/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-kube-api-access-hgx62\") pod \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " Nov 02 22:03:18.714314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:18.714301 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-plugins-config-volume\") pod \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\" (UID: \"4ebf0ea5-1a63-4c90-8870-d835e4ff1142\") " Nov 02 22:03:18.714734 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:18.714708 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-plugins-config-volume" pod "4ebf0ea5-1a63-4c90-8870-d835e4ff1142" (UID: "4ebf0ea5-1a63-4c90-8870-d835e4ff1142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:18.716355 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:18.716332 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-kube-api-access-hgx62" pod "4ebf0ea5-1a63-4c90-8870-d835e4ff1142" (UID: "4ebf0ea5-1a63-4c90-8870-d835e4ff1142"). InnerVolumeSpecName "kube-api-access-hgx62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:18.815899 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:18.815823 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:18.815899 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:18.815849 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgx62\" (UniqueName: \"kubernetes.io/projected/4ebf0ea5-1a63-4c90-8870-d835e4ff1142-kube-api-access-hgx62\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:19.345493 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.345461 4510 generic.go:356] "Generic (PLEG): container finished" podID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" containerID="9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b" exitCode=0 Nov 02 22:03:19.345665 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.345507 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" Nov 02 22:03:19.345665 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.345553 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" event={"ID":"4ebf0ea5-1a63-4c90-8870-d835e4ff1142","Type":"ContainerDied","Data":"9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b"} Nov 02 22:03:19.345665 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.345609 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv" event={"ID":"4ebf0ea5-1a63-4c90-8870-d835e4ff1142","Type":"ContainerDied","Data":"258ba8d3a40ca27f5a0e2c09dde28e97dd3c88e2980e7cbd95a6c3e0b317438f"} Nov 02 22:03:19.345665 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.345631 4510 scope.go:122] "RemoveContainer" containerID="9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b" Nov 02 22:03:19.355632 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.355608 4510 scope.go:122] "RemoveContainer" containerID="9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b" Nov 02 22:03:19.355902 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:19.355885 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b\": container with ID starting with 9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b not found: ID does not exist" containerID="9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b" Nov 02 22:03:19.355946 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.355911 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b"} err="failed to get container status \"9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b\": rpc error: code = NotFound desc = could not find container \"9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b\": container with ID starting with 9c314ff2e21f953cc930c2768f95529464d1eaa7766c5e25a1e5a8ca02c51e0b not found: ID does not exist" Nov 02 22:03:19.519279 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.519246 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv"] Nov 02 22:03:19.528828 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:19.528795 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c2dd3b-576654d767-9lckv"] Nov 02 22:03:20.158120 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:20.158081 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" path="/var/lib/kubelet/pods/4ebf0ea5-1a63-4c90-8870-d835e4ff1142/volumes" Nov 02 22:03:31.146036 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146001 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx"] Nov 02 22:03:31.146602 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146586 4510 cpu_manager.go:414] "containerMap: removing container" podUID="71a6aa31-bb91-41cd-a91b-d83d613e2275" containerName="maas-api" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146605 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a6aa31-bb91-41cd-a91b-d83d613e2275" containerName="maas-api" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146627 4510 cpu_manager.go:414] "containerMap: removing container" podUID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" containerName="payload-pre-processing" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146633 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" containerName="payload-pre-processing" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146647 4510 cpu_manager.go:414] "containerMap: removing container" podUID="76a5bef8-78a8-4e8e-9112-282d394069b4" containerName="payload-processing" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146652 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a5bef8-78a8-4e8e-9112-282d394069b4" containerName="payload-processing" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146660 4510 cpu_manager.go:414] "containerMap: removing container" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146665 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" Nov 02 22:03:31.146671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146674 4510 cpu_manager.go:414] "containerMap: removing container" podUID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" containerName="payload-pre-processing" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146682 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" containerName="payload-pre-processing" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146698 4510 cpu_manager.go:414] "containerMap: removing container" podUID="4233e721-fed9-4272-b083-700f52ffecda" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146702 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4233e721-fed9-4272-b083-700f52ffecda" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146708 4510 cpu_manager.go:414] "containerMap: removing container" podUID="70ad2a87-ef27-4f57-990d-860fd992a0a3" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146712 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ad2a87-ef27-4f57-990d-860fd992a0a3" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146718 4510 cpu_manager.go:414] "containerMap: removing container" podUID="b8b15b7e-e88e-47a2-9472-a26694c11ce9" containerName="main" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146722 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b15b7e-e88e-47a2-9472-a26694c11ce9" containerName="main" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146731 4510 cpu_manager.go:414] "containerMap: removing container" podUID="abbc8325-78a6-4d47-ad11-b255b5428937" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146735 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbc8325-78a6-4d47-ad11-b255b5428937" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146744 4510 cpu_manager.go:414] "containerMap: removing container" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146749 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146762 4510 cpu_manager.go:414] "containerMap: removing container" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146769 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146778 4510 cpu_manager.go:414] "containerMap: removing container" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146782 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146790 4510 cpu_manager.go:414] "containerMap: removing container" podUID="a5818bc3-4b3b-4012-8587-ac0c506eb11a" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146795 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5818bc3-4b3b-4012-8587-ac0c506eb11a" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146875 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59f0cae-4ae9-4841-8bca-b9d38ca799bc" containerName="payload-pre-processing" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146887 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4233e721-fed9-4272-b083-700f52ffecda" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146894 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ad2a87-ef27-4f57-990d-860fd992a0a3" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146900 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b15b7e-e88e-47a2-9472-a26694c11ce9" containerName="main" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146908 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbc8325-78a6-4d47-ad11-b255b5428937" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146918 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146924 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a38d552-966f-4ca0-8f60-be2db4e772fe" containerName="main" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146929 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="467f2a6a-2feb-43ab-b240-d909953c0cb5" containerName="istio-proxy" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146934 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5818bc3-4b3b-4012-8587-ac0c506eb11a" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146942 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a6aa31-bb91-41cd-a91b-d83d613e2275" containerName="maas-api" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146948 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebf0ea5-1a63-4c90-8870-d835e4ff1142" containerName="payload-pre-processing" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.146995 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a5bef8-78a8-4e8e-9112-282d394069b4" containerName="payload-processing" Nov 02 22:03:31.147082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.147003 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4486fa-40c4-4a43-90b4-07e436859993" containerName="main" Nov 02 22:03:31.150445 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.150424 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.198868 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.198842 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-49c06d-serving-cert\"" Nov 02 22:03:31.245455 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.245426 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx"] Nov 02 22:03:31.257038 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.257004 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.257174 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.257152 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-metrics-certs\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.257248 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.257204 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s78c\" (UniqueName: \"kubernetes.io/projected/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-kube-api-access-9s78c\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.257314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.257279 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.358858 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.358828 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-metrics-certs\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.359044 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.358877 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s78c\" (UniqueName: \"kubernetes.io/projected/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-kube-api-access-9s78c\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.359044 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.358993 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.359193 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.359176 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.359338 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:31.359317 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-serving-cert: secret "maas-api-e2e-inf-b-49c06d-serving-cert" not found Nov 02 22:03:31.359409 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:31.359398 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls podName:5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed nodeName:}" failed. No retries permitted until 2026-11-02 22:03:31.859377679 +0000 UTC m=+3037.835675444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls") pod "maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" (UID: "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed") : secret "maas-api-e2e-inf-b-49c06d-serving-cert" not found Nov 02 22:03:31.359721 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.359697 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-openshift-service-ca\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.361183 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.361159 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-metrics-certs\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.414404 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.414350 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s78c\" (UniqueName: \"kubernetes.io/projected/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-kube-api-access-9s78c\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.865761 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.865726 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:31.868239 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:31.868206 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls\") pod \"maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:32.064027 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:32.063993 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:32.437966 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:32.437891 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx"] Nov 02 22:03:32.444774 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:03:32.444745 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7f1dfb_0d83_4ecd_ab73_3b01b24fc6ed.slice/crio-fcca36abf1353781bd8cc3151936fb520d92cf8dd77db866f136ae330351d592 WatchSource:0}: Error finding container fcca36abf1353781bd8cc3151936fb520d92cf8dd77db866f136ae330351d592: Status 404 returned error can't find the container with id fcca36abf1353781bd8cc3151936fb520d92cf8dd77db866f136ae330351d592 Nov 02 22:03:33.434148 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:33.434115 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" event={"ID":"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed","Type":"ContainerStarted","Data":"d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043"} Nov 02 22:03:33.434338 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:33.434317 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:33.434387 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:33.434373 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" event={"ID":"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed","Type":"ContainerStarted","Data":"fcca36abf1353781bd8cc3151936fb520d92cf8dd77db866f136ae330351d592"} Nov 02 22:03:33.567029 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:33.566966 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" podStartSLOduration=1.6808478550000001 podStartE2EDuration="2.566947759s" totalImagesPullingTime="886.099904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:33.565768084 +0000 UTC m=+3039.542065842" watchObservedRunningTime="2026-11-02 22:03:33.566947759 +0000 UTC m=+3039.543245537" Nov 02 22:03:35.636616 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:35.636582 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26"] Nov 02 22:03:35.637029 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:35.636783 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" podUID="8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" containerName="maas-api" containerID="cri-o://d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81" gracePeriod=30 Nov 02 22:03:35.901009 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:35.900949 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 22:03:36.011945 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.011910 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls\") pod \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " Nov 02 22:03:36.012139 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.012046 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-kube-api-access-wffmq\" (UniqueName: \"kubernetes.io/projected/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-kube-api-access-wffmq\") pod \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " Nov 02 22:03:36.012139 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.012096 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-metrics-certs\") pod \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " Nov 02 22:03:36.012268 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.012173 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-openshift-service-ca\") pod \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\" (UID: \"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e\") " Nov 02 22:03:36.012949 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.012916 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-openshift-service-ca" pod "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" (UID: "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:03:36.022637 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.022462 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls" pod "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" (UID: "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:36.022637 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.022595 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-metrics-certs" pod "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" (UID: "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:03:36.027956 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.027931 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-kube-api-access-wffmq" pod "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" (UID: "8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e"). InnerVolumeSpecName "kube-api-access-wffmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:03:36.113493 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.113464 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:36.113493 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.113491 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wffmq\" (UniqueName: \"kubernetes.io/projected/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-kube-api-access-wffmq\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:36.113493 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.113501 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:36.113734 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.113510 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:03:36.452541 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.452430 4510 generic.go:356] "Generic (PLEG): container finished" podID="8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" containerID="d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81" exitCode=0 Nov 02 22:03:36.452541 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.452479 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" Nov 02 22:03:36.452759 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.452548 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" event={"ID":"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e","Type":"ContainerDied","Data":"d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81"} Nov 02 22:03:36.452759 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.452601 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26" event={"ID":"8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e","Type":"ContainerDied","Data":"de1b3b35ff84729b98e9f1303fcbdc1b0315a9df95b8dad3226d7080387f8127"} Nov 02 22:03:36.452759 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.452625 4510 scope.go:122] "RemoveContainer" containerID="d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81" Nov 02 22:03:36.466386 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.466363 4510 scope.go:122] "RemoveContainer" containerID="d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81" Nov 02 22:03:36.466780 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:36.466752 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81\": container with ID starting with d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81 not found: ID does not exist" containerID="d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81" Nov 02 22:03:36.466904 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.466795 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81"} err="failed to get container status \"d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81\": rpc error: code = NotFound desc = could not find container \"d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81\": container with ID starting with d94865ce59f70c22cf9092f7080404decb0274b026b195d80ab6cdc46ddcfb81 not found: ID does not exist" Nov 02 22:03:36.604852 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.604815 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26"] Nov 02 22:03:36.607596 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:36.607566 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1243c-6558d88d68-ghf26"] Nov 02 22:03:38.155778 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:38.155744 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" path="/var/lib/kubelet/pods/8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e/volumes" Nov 02 22:03:39.445378 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:39.445350 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:03:45.531655 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:03:45.531618 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:03:47.086565 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:47.086532 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:03:56.922084 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:03:56.922052 4510 scope.go:122] "RemoveContainer" containerID="52c34f1617a8c60ba7bbdb62c76bc91bb0e982dd8fd3ab58e204dbdede795cad" Nov 02 22:04:01.014378 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.014346 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44"] Nov 02 22:04:01.014854 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.014582 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" podUID="d29a6515-d0a5-4982-b781-1459c27f6d8b" containerName="maas-api" containerID="cri-o://989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e" gracePeriod=30 Nov 02 22:04:01.101370 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.101327 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" podUID="d29a6515-d0a5-4982-b781-1459c27f6d8b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.137:8443/health\": dial tcp 10.134.0.137:8443: connect: connection refused" Nov 02 22:04:01.273057 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.272990 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 22:04:01.398957 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.398924 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29a6515-d0a5-4982-b781-1459c27f6d8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a6515-d0a5-4982-b781-1459c27f6d8b-openshift-service-ca\") pod \"d29a6515-d0a5-4982-b781-1459c27f6d8b\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " Nov 02 22:04:01.399142 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.398968 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls\") pod \"d29a6515-d0a5-4982-b781-1459c27f6d8b\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " Nov 02 22:04:01.399142 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.399111 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-metrics-certs\") pod \"d29a6515-d0a5-4982-b781-1459c27f6d8b\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " Nov 02 22:04:01.399358 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.399339 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29a6515-d0a5-4982-b781-1459c27f6d8b-kube-api-access-ptpzs\" (UniqueName: \"kubernetes.io/projected/d29a6515-d0a5-4982-b781-1459c27f6d8b-kube-api-access-ptpzs\") pod \"d29a6515-d0a5-4982-b781-1459c27f6d8b\" (UID: \"d29a6515-d0a5-4982-b781-1459c27f6d8b\") " Nov 02 22:04:01.399564 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.399387 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29a6515-d0a5-4982-b781-1459c27f6d8b-openshift-service-ca" pod "d29a6515-d0a5-4982-b781-1459c27f6d8b" (UID: "d29a6515-d0a5-4982-b781-1459c27f6d8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:01.400049 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.400021 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29a6515-d0a5-4982-b781-1459c27f6d8b-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:01.401502 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.401477 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29a6515-d0a5-4982-b781-1459c27f6d8b-kube-api-access-ptpzs" pod "d29a6515-d0a5-4982-b781-1459c27f6d8b" (UID: "d29a6515-d0a5-4982-b781-1459c27f6d8b"). InnerVolumeSpecName "kube-api-access-ptpzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:01.401709 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.401685 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls" pod "d29a6515-d0a5-4982-b781-1459c27f6d8b" (UID: "d29a6515-d0a5-4982-b781-1459c27f6d8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:01.401825 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.401790 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-metrics-certs" pod "d29a6515-d0a5-4982-b781-1459c27f6d8b" (UID: "d29a6515-d0a5-4982-b781-1459c27f6d8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:01.501511 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.501465 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptpzs\" (UniqueName: \"kubernetes.io/projected/d29a6515-d0a5-4982-b781-1459c27f6d8b-kube-api-access-ptpzs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:01.501511 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.501506 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:01.501511 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.501547 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d29a6515-d0a5-4982-b781-1459c27f6d8b-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:01.601946 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.601864 4510 generic.go:356] "Generic (PLEG): container finished" podID="d29a6515-d0a5-4982-b781-1459c27f6d8b" containerID="989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e" exitCode=0 Nov 02 22:04:01.601946 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.601910 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" Nov 02 22:04:01.602133 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.601949 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" event={"ID":"d29a6515-d0a5-4982-b781-1459c27f6d8b","Type":"ContainerDied","Data":"989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e"} Nov 02 22:04:01.602133 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.602004 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44" event={"ID":"d29a6515-d0a5-4982-b781-1459c27f6d8b","Type":"ContainerDied","Data":"dc93ab6a96eba0f2d774a67a798ba257af0bab1fb3d218c55005408374c71fdf"} Nov 02 22:04:01.602133 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.602030 4510 scope.go:122] "RemoveContainer" containerID="989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e" Nov 02 22:04:01.612050 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.612034 4510 scope.go:122] "RemoveContainer" containerID="989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e" Nov 02 22:04:01.612288 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:01.612270 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e\": container with ID starting with 989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e not found: ID does not exist" containerID="989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e" Nov 02 22:04:01.612338 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.612297 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e"} err="failed to get container status \"989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e\": rpc error: code = NotFound desc = could not find container \"989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e\": container with ID starting with 989bc5e3911269f0f8da7a62e51d21e93f20c3058724b8ba26352e5cdaa1a18e not found: ID does not exist" Nov 02 22:04:01.754214 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.754185 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44"] Nov 02 22:04:01.756423 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:01.756385 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bcb857-7d9ffdf85d-6lg44"] Nov 02 22:04:02.155373 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:02.155337 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29a6515-d0a5-4982-b781-1459c27f6d8b" path="/var/lib/kubelet/pods/d29a6515-d0a5-4982-b781-1459c27f6d8b/volumes" Nov 02 22:04:11.765407 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:11.765376 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6"] Nov 02 22:04:11.765975 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:11.765622 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" podUID="5e57a579-8556-4966-afdd-ccf6f0e38c4e" containerName="maas-api" containerID="cri-o://83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0" gracePeriod=30 Nov 02 22:04:12.036332 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.036308 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 22:04:12.115034 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.115011 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls\") pod \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " Nov 02 22:04:12.115173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.115066 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e57a579-8556-4966-afdd-ccf6f0e38c4e-kube-api-access-2jdj8\" (UniqueName: \"kubernetes.io/projected/5e57a579-8556-4966-afdd-ccf6f0e38c4e-kube-api-access-2jdj8\") pod \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " Nov 02 22:04:12.115173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.115102 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e57a579-8556-4966-afdd-ccf6f0e38c4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57a579-8556-4966-afdd-ccf6f0e38c4e-openshift-service-ca\") pod \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " Nov 02 22:04:12.115173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.115126 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-metrics-certs\") pod \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\" (UID: \"5e57a579-8556-4966-afdd-ccf6f0e38c4e\") " Nov 02 22:04:12.115442 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.115414 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e57a579-8556-4966-afdd-ccf6f0e38c4e-openshift-service-ca" pod "5e57a579-8556-4966-afdd-ccf6f0e38c4e" (UID: "5e57a579-8556-4966-afdd-ccf6f0e38c4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:12.115588 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.115543 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e57a579-8556-4966-afdd-ccf6f0e38c4e-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:12.117191 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.117162 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e57a579-8556-4966-afdd-ccf6f0e38c4e-kube-api-access-2jdj8" pod "5e57a579-8556-4966-afdd-ccf6f0e38c4e" (UID: "5e57a579-8556-4966-afdd-ccf6f0e38c4e"). InnerVolumeSpecName "kube-api-access-2jdj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:12.117191 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.117180 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-metrics-certs" pod "5e57a579-8556-4966-afdd-ccf6f0e38c4e" (UID: "5e57a579-8556-4966-afdd-ccf6f0e38c4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:12.117309 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.117197 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls" pod "5e57a579-8556-4966-afdd-ccf6f0e38c4e" (UID: "5e57a579-8556-4966-afdd-ccf6f0e38c4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:12.216830 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.216802 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:12.216830 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.216826 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdj8\" (UniqueName: \"kubernetes.io/projected/5e57a579-8556-4966-afdd-ccf6f0e38c4e-kube-api-access-2jdj8\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:12.216830 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.216836 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e57a579-8556-4966-afdd-ccf6f0e38c4e-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:12.673071 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.673029 4510 generic.go:356] "Generic (PLEG): container finished" podID="5e57a579-8556-4966-afdd-ccf6f0e38c4e" containerID="83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0" exitCode=0 Nov 02 22:04:12.673277 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.673086 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" event={"ID":"5e57a579-8556-4966-afdd-ccf6f0e38c4e","Type":"ContainerDied","Data":"83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0"} Nov 02 22:04:12.673277 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.673103 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" Nov 02 22:04:12.673277 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.673157 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6" event={"ID":"5e57a579-8556-4966-afdd-ccf6f0e38c4e","Type":"ContainerDied","Data":"2af69215a468cc93ea6fae11c007ed89131a84c0cc524e877d3e34b4bb4fd155"} Nov 02 22:04:12.673277 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.673242 4510 scope.go:122] "RemoveContainer" containerID="83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0" Nov 02 22:04:12.684948 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.684923 4510 scope.go:122] "RemoveContainer" containerID="83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0" Nov 02 22:04:12.685242 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:12.685222 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0\": container with ID starting with 83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0 not found: ID does not exist" containerID="83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0" Nov 02 22:04:12.685317 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.685251 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0"} err="failed to get container status \"83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0\": rpc error: code = NotFound desc = could not find container \"83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0\": container with ID starting with 83ea2031d4c7df13683fe449930292b106ec71d85b9ec95eec038a607fa4a1f0 not found: ID does not exist" Nov 02 22:04:12.826107 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.826080 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6"] Nov 02 22:04:12.872635 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:12.872598 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f22129-5fc776ff75-lxsg6"] Nov 02 22:04:14.154220 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:14.154184 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e57a579-8556-4966-afdd-ccf6f0e38c4e" path="/var/lib/kubelet/pods/5e57a579-8556-4966-afdd-ccf6f0e38c4e/volumes" Nov 02 22:04:14.984494 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:14.984463 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f"] Nov 02 22:04:15.531544 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:15.531493 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:04:17.703679 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:17.703646 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw"] Nov 02 22:04:17.704185 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:17.703882 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" podUID="110c5f36-3efa-4fdb-b8fc-4eb7121a4765" containerName="maas-api" containerID="cri-o://312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8" gracePeriod=30 Nov 02 22:04:17.965257 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:17.965196 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 22:04:18.081159 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.081121 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls\") pod \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " Nov 02 22:04:18.081344 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.081216 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-openshift-service-ca\") pod \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " Nov 02 22:04:18.081344 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.081270 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-metrics-certs\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-metrics-certs\") pod \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " Nov 02 22:04:18.081476 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.081380 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-kube-api-access-ldpwv\" (UniqueName: \"kubernetes.io/projected/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-kube-api-access-ldpwv\") pod \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\" (UID: \"110c5f36-3efa-4fdb-b8fc-4eb7121a4765\") " Nov 02 22:04:18.082731 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.082703 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-openshift-service-ca" pod "110c5f36-3efa-4fdb-b8fc-4eb7121a4765" (UID: "110c5f36-3efa-4fdb-b8fc-4eb7121a4765"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:18.083556 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.083503 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls" pod "110c5f36-3efa-4fdb-b8fc-4eb7121a4765" (UID: "110c5f36-3efa-4fdb-b8fc-4eb7121a4765"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:18.083754 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.083730 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-kube-api-access-ldpwv" pod "110c5f36-3efa-4fdb-b8fc-4eb7121a4765" (UID: "110c5f36-3efa-4fdb-b8fc-4eb7121a4765"). InnerVolumeSpecName "kube-api-access-ldpwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:18.083919 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.083903 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-metrics-certs" pod "110c5f36-3efa-4fdb-b8fc-4eb7121a4765" (UID: "110c5f36-3efa-4fdb-b8fc-4eb7121a4765"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:18.182671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.182640 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:18.182671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.182667 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:18.182671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.182677 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:18.182873 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.182685 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldpwv\" (UniqueName: \"kubernetes.io/projected/110c5f36-3efa-4fdb-b8fc-4eb7121a4765-kube-api-access-ldpwv\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:18.528913 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.528874 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:04:18.708847 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.708810 4510 generic.go:356] "Generic (PLEG): container finished" podID="110c5f36-3efa-4fdb-b8fc-4eb7121a4765" containerID="312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8" exitCode=0 Nov 02 22:04:18.709340 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.708854 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" Nov 02 22:04:18.709340 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.708895 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" event={"ID":"110c5f36-3efa-4fdb-b8fc-4eb7121a4765","Type":"ContainerDied","Data":"312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8"} Nov 02 22:04:18.709340 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.708947 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw" event={"ID":"110c5f36-3efa-4fdb-b8fc-4eb7121a4765","Type":"ContainerDied","Data":"e6396272a2288a566657c6c8f83cbeff95e6100455118217cceb2dd5a71662fc"} Nov 02 22:04:18.709340 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.708973 4510 scope.go:122] "RemoveContainer" containerID="312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8" Nov 02 22:04:18.719094 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.719075 4510 scope.go:122] "RemoveContainer" containerID="312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8" Nov 02 22:04:18.719382 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:18.719359 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8\": container with ID starting with 312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8 not found: ID does not exist" containerID="312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8" Nov 02 22:04:18.719433 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.719387 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8"} err="failed to get container status \"312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8\": rpc error: code = NotFound desc = could not find container \"312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8\": container with ID starting with 312a5ffa7ff2f9713a8a542c222e9c869d2603cb06496d10c946a25697bf31f8 not found: ID does not exist" Nov 02 22:04:18.861269 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.861183 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw"] Nov 02 22:04:18.864898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:18.864872 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c2dd3b-6c9589fcb-zs4rw"] Nov 02 22:04:20.154971 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:20.154935 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110c5f36-3efa-4fdb-b8fc-4eb7121a4765" path="/var/lib/kubelet/pods/110c5f36-3efa-4fdb-b8fc-4eb7121a4765/volumes" Nov 02 22:04:23.531307 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:23.531270 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467"] Nov 02 22:04:23.719801 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:23.719765 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84"] Nov 02 22:04:23.719964 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:23.719945 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" podUID="1c41d7f8-ea78-4117-86f8-2187e5724ba7" containerName="maas-api" containerID="cri-o://c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8" gracePeriod=30 Nov 02 22:04:23.983918 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:23.983896 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 22:04:24.057151 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.057124 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-metrics-certs\") pod \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " Nov 02 22:04:24.057318 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.057232 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c41d7f8-ea78-4117-86f8-2187e5724ba7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d7f8-ea78-4117-86f8-2187e5724ba7-openshift-service-ca\") pod \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " Nov 02 22:04:24.057318 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.057252 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls\") pod \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " Nov 02 22:04:24.057436 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.057318 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c41d7f8-ea78-4117-86f8-2187e5724ba7-kube-api-access-5cf82\" (UniqueName: \"kubernetes.io/projected/1c41d7f8-ea78-4117-86f8-2187e5724ba7-kube-api-access-5cf82\") pod \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\" (UID: \"1c41d7f8-ea78-4117-86f8-2187e5724ba7\") " Nov 02 22:04:24.057670 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.057643 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c41d7f8-ea78-4117-86f8-2187e5724ba7-openshift-service-ca" pod "1c41d7f8-ea78-4117-86f8-2187e5724ba7" (UID: "1c41d7f8-ea78-4117-86f8-2187e5724ba7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:24.059259 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.059225 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-metrics-certs" pod "1c41d7f8-ea78-4117-86f8-2187e5724ba7" (UID: "1c41d7f8-ea78-4117-86f8-2187e5724ba7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:24.059377 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.059322 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls" pod "1c41d7f8-ea78-4117-86f8-2187e5724ba7" (UID: "1c41d7f8-ea78-4117-86f8-2187e5724ba7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:24.059432 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.059412 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c41d7f8-ea78-4117-86f8-2187e5724ba7-kube-api-access-5cf82" pod "1c41d7f8-ea78-4117-86f8-2187e5724ba7" (UID: "1c41d7f8-ea78-4117-86f8-2187e5724ba7"). InnerVolumeSpecName "kube-api-access-5cf82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:24.159210 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.159180 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cf82\" (UniqueName: \"kubernetes.io/projected/1c41d7f8-ea78-4117-86f8-2187e5724ba7-kube-api-access-5cf82\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:24.159210 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.159211 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:24.159458 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.159227 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c41d7f8-ea78-4117-86f8-2187e5724ba7-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:24.159458 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.159241 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c41d7f8-ea78-4117-86f8-2187e5724ba7-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:24.748680 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.748644 4510 generic.go:356] "Generic (PLEG): container finished" podID="1c41d7f8-ea78-4117-86f8-2187e5724ba7" containerID="c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8" exitCode=0 Nov 02 22:04:24.749090 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.748724 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" Nov 02 22:04:24.749090 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.748725 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" event={"ID":"1c41d7f8-ea78-4117-86f8-2187e5724ba7","Type":"ContainerDied","Data":"c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8"} Nov 02 22:04:24.749090 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.748865 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84" event={"ID":"1c41d7f8-ea78-4117-86f8-2187e5724ba7","Type":"ContainerDied","Data":"6fcd326c3b8d90286a0dfd80deb074566707ebbf0c2d1c1fed3ab3921a79cf2d"} Nov 02 22:04:24.749090 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.748887 4510 scope.go:122] "RemoveContainer" containerID="c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8" Nov 02 22:04:24.761838 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.761812 4510 scope.go:122] "RemoveContainer" containerID="c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8" Nov 02 22:04:24.762370 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:24.762345 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8\": container with ID starting with c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8 not found: ID does not exist" containerID="c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8" Nov 02 22:04:24.762451 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.762383 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8"} err="failed to get container status \"c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8\": rpc error: code = NotFound desc = could not find container \"c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8\": container with ID starting with c2d380d786f4e233ba015d2f4328c851d59dd8f59dad0e80ae13d2ac3c3248c8 not found: ID does not exist" Nov 02 22:04:24.915506 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.915464 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84"] Nov 02 22:04:24.959418 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:24.959378 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70d237-788446b84d-6hw84"] Nov 02 22:04:26.153419 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:26.153387 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c41d7f8-ea78-4117-86f8-2187e5724ba7" path="/var/lib/kubelet/pods/1c41d7f8-ea78-4117-86f8-2187e5724ba7/volumes" Nov 02 22:04:26.178561 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:26.178532 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf"] Nov 02 22:04:27.587438 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.587404 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj"] Nov 02 22:04:27.587953 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.587938 4510 cpu_manager.go:414] "containerMap: removing container" podUID="5e57a579-8556-4966-afdd-ccf6f0e38c4e" containerName="maas-api" Nov 02 22:04:27.587998 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.587954 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e57a579-8556-4966-afdd-ccf6f0e38c4e" containerName="maas-api" Nov 02 22:04:27.587998 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.587974 4510 cpu_manager.go:414] "containerMap: removing container" podUID="110c5f36-3efa-4fdb-b8fc-4eb7121a4765" containerName="maas-api" Nov 02 22:04:27.587998 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.587979 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110c5f36-3efa-4fdb-b8fc-4eb7121a4765" containerName="maas-api" Nov 02 22:04:27.587998 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.587988 4510 cpu_manager.go:414] "containerMap: removing container" podUID="8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" containerName="maas-api" Nov 02 22:04:27.587998 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.587995 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588018 4510 cpu_manager.go:414] "containerMap: removing container" podUID="1c41d7f8-ea78-4117-86f8-2187e5724ba7" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588022 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41d7f8-ea78-4117-86f8-2187e5724ba7" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588033 4510 cpu_manager.go:414] "containerMap: removing container" podUID="d29a6515-d0a5-4982-b781-1459c27f6d8b" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588038 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29a6515-d0a5-4982-b781-1459c27f6d8b" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588124 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41d7f8-ea78-4117-86f8-2187e5724ba7" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588136 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29a6515-d0a5-4982-b781-1459c27f6d8b" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588145 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e57a579-8556-4966-afdd-ccf6f0e38c4e" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588155 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="110c5f36-3efa-4fdb-b8fc-4eb7121a4765" containerName="maas-api" Nov 02 22:04:27.588173 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.588163 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c94fd2c-6d5e-4366-82ee-9d46ab7f3b6e" containerName="maas-api" Nov 02 22:04:27.591236 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.591218 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.664863 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.664830 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj"] Nov 02 22:04:27.689353 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.689329 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c4a6b9\"/\"test-model-c4a6b9-kserve-self-signed-certs\"" Nov 02 22:04:27.689472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.689358 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c4a6b9\"/\"openshift-service-ca.crt\"" Nov 02 22:04:27.689472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.689372 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c4a6b9\"/\"kube-root-ca.crt\"" Nov 02 22:04:27.689472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.689382 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c4a6b9\"/\"default-dockercfg-mvprx\"" Nov 02 22:04:27.698030 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.698010 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47nr9\" (UniqueName: \"kubernetes.io/projected/42d51387-efaa-4d06-b813-25b70ebbf594-kube-api-access-47nr9\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.698124 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.698046 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-tmp-dir\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.698124 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.698075 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-home\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.698238 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.698220 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-dshm\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.698285 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.698268 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-model-cache\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.698383 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.698367 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d51387-efaa-4d06-b813-25b70ebbf594-tls-certs\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799294 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799265 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nr9\" (UniqueName: \"kubernetes.io/projected/42d51387-efaa-4d06-b813-25b70ebbf594-kube-api-access-47nr9\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799482 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799303 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-tmp-dir\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799482 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799327 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-home\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799482 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799384 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-dshm\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799482 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799433 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-model-cache\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799482 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799470 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d51387-efaa-4d06-b813-25b70ebbf594-tls-certs\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799807 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799783 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-tmp-dir\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799865 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799839 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-home\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.799913 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.799895 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-model-cache\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.801683 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.801662 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-dshm\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.802020 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.802003 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d51387-efaa-4d06-b813-25b70ebbf594-tls-certs\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.852854 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.852797 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nr9\" (UniqueName: \"kubernetes.io/projected/42d51387-efaa-4d06-b813-25b70ebbf594-kube-api-access-47nr9\") pod \"test-model-c4a6b9-kserve-74bd4b85-qnkbj\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:27.903371 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:27.903345 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:28.272782 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:28.272752 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj"] Nov 02 22:04:28.279844 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:04:28.279818 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42d51387_efaa_4d06_b813_25b70ebbf594.slice/crio-6dfb58b4a91e83a41bbfc5dc753e91a1e76d5628cd3cfce3d56b959cdb795cd3 WatchSource:0}: Error finding container 6dfb58b4a91e83a41bbfc5dc753e91a1e76d5628cd3cfce3d56b959cdb795cd3: Status 404 returned error can't find the container with id 6dfb58b4a91e83a41bbfc5dc753e91a1e76d5628cd3cfce3d56b959cdb795cd3 Nov 02 22:04:28.532180 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:28.532102 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" podUID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" containerName="payload-processing" containerID="cri-o://0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b" gracePeriod=25 Nov 02 22:04:28.772710 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:28.772678 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" event={"ID":"42d51387-efaa-4d06-b813-25b70ebbf594","Type":"ContainerStarted","Data":"9b91dfd535e2708e1099733d4cca14bb27cdf986008e805a35362c735c1901fb"} Nov 02 22:04:28.773101 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:28.772720 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" event={"ID":"42d51387-efaa-4d06-b813-25b70ebbf594","Type":"ContainerStarted","Data":"6dfb58b4a91e83a41bbfc5dc753e91a1e76d5628cd3cfce3d56b959cdb795cd3"} Nov 02 22:04:28.879388 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:28.879342 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" podStartSLOduration=1.879325168 podStartE2EDuration="1.879325168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:28.877328995 +0000 UTC m=+3094.853626772" watchObservedRunningTime="2026-11-02 22:04:28.879325168 +0000 UTC m=+3094.855622944" Nov 02 22:04:28.903957 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:28.903934 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:30.005303 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.005264 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" containerID="cri-o://78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42" gracePeriod=45 Nov 02 22:04:30.150938 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.150898 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused" Nov 02 22:04:30.261155 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.261089 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 22:04:30.315505 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.315465 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" podUID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:30.327844 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.327818 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baea62ff-7e44-4e92-83b4-018d05ecc226-tls-certs\" (UniqueName: \"kubernetes.io/secret/baea62ff-7e44-4e92-83b4-018d05ecc226-tls-certs\") pod \"baea62ff-7e44-4e92-83b4-018d05ecc226\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " Nov 02 22:04:30.327982 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.327889 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baea62ff-7e44-4e92-83b4-018d05ecc226-kube-api-access-9bd2j\" (UniqueName: \"kubernetes.io/projected/baea62ff-7e44-4e92-83b4-018d05ecc226-kube-api-access-9bd2j\") pod \"baea62ff-7e44-4e92-83b4-018d05ecc226\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " Nov 02 22:04:30.328046 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.327994 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-tmp-dir\") pod \"baea62ff-7e44-4e92-83b4-018d05ecc226\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " Nov 02 22:04:30.328107 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328081 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-dshm\") pod \"baea62ff-7e44-4e92-83b4-018d05ecc226\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " Nov 02 22:04:30.328163 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328124 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-home\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-home\") pod \"baea62ff-7e44-4e92-83b4-018d05ecc226\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " Nov 02 22:04:30.328216 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328161 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-model-cache\") pod \"baea62ff-7e44-4e92-83b4-018d05ecc226\" (UID: \"baea62ff-7e44-4e92-83b4-018d05ecc226\") " Nov 02 22:04:30.328281 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328253 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-tmp-dir" pod "baea62ff-7e44-4e92-83b4-018d05ecc226" (UID: "baea62ff-7e44-4e92-83b4-018d05ecc226"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.328435 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328410 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-home" pod "baea62ff-7e44-4e92-83b4-018d05ecc226" (UID: "baea62ff-7e44-4e92-83b4-018d05ecc226"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.328497 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328444 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-model-cache" pod "baea62ff-7e44-4e92-83b4-018d05ecc226" (UID: "baea62ff-7e44-4e92-83b4-018d05ecc226"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.328630 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328609 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.328630 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328629 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.328764 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.328638 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.330145 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.330118 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baea62ff-7e44-4e92-83b4-018d05ecc226-kube-api-access-9bd2j" pod "baea62ff-7e44-4e92-83b4-018d05ecc226" (UID: "baea62ff-7e44-4e92-83b4-018d05ecc226"). InnerVolumeSpecName "kube-api-access-9bd2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:30.330261 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.330157 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baea62ff-7e44-4e92-83b4-018d05ecc226-tls-certs" pod "baea62ff-7e44-4e92-83b4-018d05ecc226" (UID: "baea62ff-7e44-4e92-83b4-018d05ecc226"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:30.330261 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.330212 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-dshm" pod "baea62ff-7e44-4e92-83b4-018d05ecc226" (UID: "baea62ff-7e44-4e92-83b4-018d05ecc226"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:30.430034 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.430003 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baea62ff-7e44-4e92-83b4-018d05ecc226-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.430034 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.430031 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baea62ff-7e44-4e92-83b4-018d05ecc226-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.430034 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.430042 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bd2j\" (UniqueName: \"kubernetes.io/projected/baea62ff-7e44-4e92-83b4-018d05ecc226-kube-api-access-9bd2j\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:30.785077 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.785038 4510 generic.go:356] "Generic (PLEG): container finished" podID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerID="78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42" exitCode=0 Nov 02 22:04:30.785077 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.785074 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" Nov 02 22:04:30.785331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.785124 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" event={"ID":"baea62ff-7e44-4e92-83b4-018d05ecc226","Type":"ContainerDied","Data":"78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42"} Nov 02 22:04:30.785331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.785164 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f" event={"ID":"baea62ff-7e44-4e92-83b4-018d05ecc226","Type":"ContainerDied","Data":"8c8f5027e76fe0d60f0032929e17d978d0de6e424be5a85f4ef7ee2482372662"} Nov 02 22:04:30.785331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.785180 4510 scope.go:122] "RemoveContainer" containerID="78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42" Nov 02 22:04:30.795612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.795592 4510 scope.go:122] "RemoveContainer" containerID="78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42" Nov 02 22:04:30.795881 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:30.795860 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42\": container with ID starting with 78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42 not found: ID does not exist" containerID="78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42" Nov 02 22:04:30.795975 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.795885 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42"} err="failed to get container status \"78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42\": rpc error: code = NotFound desc = could not find container \"78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42\": container with ID starting with 78bbeda2b99c3296646e4f9fce7ef88dfc837ef0d046a49662febc669e29de42 not found: ID does not exist" Nov 02 22:04:30.937783 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.937753 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f"] Nov 02 22:04:30.940056 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:30.940034 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9a6cf2/distinct-2-w3-9a6cf2-kserve-7f96c4dbcb-97z8f"] Nov 02 22:04:32.153360 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:32.153322 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" path="/var/lib/kubelet/pods/baea62ff-7e44-4e92-83b4-018d05ecc226/volumes" Nov 02 22:04:37.904277 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:37.904195 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:37.916643 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:37.916618 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:37.926740 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:37.926718 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj"] Nov 02 22:04:38.840033 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:38.839997 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:04:40.315216 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:40.315177 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" podUID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:04:41.199318 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.199275 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" containerID="cri-o://3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f" gracePeriod=45 Nov 02 22:04:41.464426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.464369 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 22:04:41.557064 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557031 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-home\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-home\") pod \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " Nov 02 22:04:41.557211 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557131 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tmp-dir\") pod \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " Nov 02 22:04:41.557211 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557161 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-dshm\") pod \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " Nov 02 22:04:41.557314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557237 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tls-certs\") pod \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " Nov 02 22:04:41.557314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557273 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-model-cache\") pod \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " Nov 02 22:04:41.557428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557317 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85f72e4-bab3-4754-8f49-463cc0c4daa9-kube-api-access-9zc6v\" (UniqueName: \"kubernetes.io/projected/c85f72e4-bab3-4754-8f49-463cc0c4daa9-kube-api-access-9zc6v\") pod \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\" (UID: \"c85f72e4-bab3-4754-8f49-463cc0c4daa9\") " Nov 02 22:04:41.557428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557346 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-home" pod "c85f72e4-bab3-4754-8f49-463cc0c4daa9" (UID: "c85f72e4-bab3-4754-8f49-463cc0c4daa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.557428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557397 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tmp-dir" pod "c85f72e4-bab3-4754-8f49-463cc0c4daa9" (UID: "c85f72e4-bab3-4754-8f49-463cc0c4daa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.557611 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557565 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-model-cache" pod "c85f72e4-bab3-4754-8f49-463cc0c4daa9" (UID: "c85f72e4-bab3-4754-8f49-463cc0c4daa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.557778 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557759 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.557857 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557779 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.557857 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.557789 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.559566 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.559538 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-dshm" pod "c85f72e4-bab3-4754-8f49-463cc0c4daa9" (UID: "c85f72e4-bab3-4754-8f49-463cc0c4daa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:41.559686 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.559570 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tls-certs" pod "c85f72e4-bab3-4754-8f49-463cc0c4daa9" (UID: "c85f72e4-bab3-4754-8f49-463cc0c4daa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:41.559686 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.559582 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85f72e4-bab3-4754-8f49-463cc0c4daa9-kube-api-access-9zc6v" pod "c85f72e4-bab3-4754-8f49-463cc0c4daa9" (UID: "c85f72e4-bab3-4754-8f49-463cc0c4daa9"). InnerVolumeSpecName "kube-api-access-9zc6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:41.658831 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.658795 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c85f72e4-bab3-4754-8f49-463cc0c4daa9-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.658831 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.658832 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c85f72e4-bab3-4754-8f49-463cc0c4daa9-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.659035 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.658850 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zc6v\" (UniqueName: \"kubernetes.io/projected/c85f72e4-bab3-4754-8f49-463cc0c4daa9-kube-api-access-9zc6v\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:41.846640 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.846557 4510 generic.go:356] "Generic (PLEG): container finished" podID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerID="3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f" exitCode=0 Nov 02 22:04:41.846640 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.846622 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" Nov 02 22:04:41.846868 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.846642 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" event={"ID":"c85f72e4-bab3-4754-8f49-463cc0c4daa9","Type":"ContainerDied","Data":"3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f"} Nov 02 22:04:41.846868 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.846687 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf" event={"ID":"c85f72e4-bab3-4754-8f49-463cc0c4daa9","Type":"ContainerDied","Data":"7cb397aeab74815a37eee894a677fc696dfbae80ab6e367b8d272eb5189acb2b"} Nov 02 22:04:41.846868 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.846707 4510 scope.go:122] "RemoveContainer" containerID="3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f" Nov 02 22:04:41.857335 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.857317 4510 scope.go:122] "RemoveContainer" containerID="3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f" Nov 02 22:04:41.857618 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:41.857598 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f\": container with ID starting with 3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f not found: ID does not exist" containerID="3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f" Nov 02 22:04:41.857683 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:41.857627 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f"} err="failed to get container status \"3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f\": rpc error: code = NotFound desc = could not find container \"3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f\": container with ID starting with 3e4e068f5c9b8522f8982142514d35a961e7de3b2070126cd6989fb9041a7c4f not found: ID does not exist" Nov 02 22:04:42.000425 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:42.000389 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf"] Nov 02 22:04:42.046154 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:42.046124 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4d9ea0/distinct-w2-4d9ea0-kserve-65779867cb-x6jgf"] Nov 02 22:04:42.153953 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:42.153923 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" path="/var/lib/kubelet/pods/c85f72e4-bab3-4754-8f49-463cc0c4daa9/volumes" Nov 02 22:04:45.533442 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:45.533404 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:04:48.784954 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.784931 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 22:04:48.894625 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.894587 4510 generic.go:356] "Generic (PLEG): container finished" podID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" containerID="0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b" exitCode=0 Nov 02 22:04:48.894795 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.894664 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" Nov 02 22:04:48.894795 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.894673 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" event={"ID":"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544","Type":"ContainerDied","Data":"0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b"} Nov 02 22:04:48.894795 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.894723 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467" event={"ID":"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544","Type":"ContainerDied","Data":"4f027ae5d8bef6dc375f1c1583106515c50e3849139a36db35a0df5799273597"} Nov 02 22:04:48.894795 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.894754 4510 scope.go:122] "RemoveContainer" containerID="0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b" Nov 02 22:04:48.904905 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.904880 4510 scope.go:122] "RemoveContainer" containerID="0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b" Nov 02 22:04:48.905163 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:48.905144 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b\": container with ID starting with 0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b not found: ID does not exist" containerID="0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b" Nov 02 22:04:48.905218 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.905178 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b"} err="failed to get container status \"0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b\": rpc error: code = NotFound desc = could not find container \"0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b\": container with ID starting with 0ecef7881fd99f010c9af722df9a31e3b92e176c9311b9d4033fee5034101d8b not found: ID does not exist" Nov 02 22:04:48.933869 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.933848 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-plugins-config-volume\") pod \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " Nov 02 22:04:48.933987 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.933902 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-kube-api-access-vnsj7\" (UniqueName: \"kubernetes.io/projected/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-kube-api-access-vnsj7\") pod \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\" (UID: \"9b30ae4c-6c6e-4975-9c84-6d76ba5c5544\") " Nov 02 22:04:48.934187 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.934167 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-plugins-config-volume" pod "9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" (UID: "9b30ae4c-6c6e-4975-9c84-6d76ba5c5544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:48.934401 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.934388 4510 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-plugins-config-volume\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:48.935767 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:48.935747 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-kube-api-access-vnsj7" pod "9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" (UID: "9b30ae4c-6c6e-4975-9c84-6d76ba5c5544"). InnerVolumeSpecName "kube-api-access-vnsj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:49.035645 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:49.035612 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsj7\" (UniqueName: \"kubernetes.io/projected/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544-kube-api-access-vnsj7\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:49.305515 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:49.305483 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467"] Nov 02 22:04:49.351565 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:49.351531 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9a6cf2-5f5bb8bd95-48467"] Nov 02 22:04:50.152576 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:50.152541 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" path="/var/lib/kubelet/pods/9b30ae4c-6c6e-4975-9c84-6d76ba5c5544/volumes" Nov 02 22:04:52.946785 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:52.946727 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" containerID="cri-o://85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6" gracePeriod=45 Nov 02 22:04:53.208025 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.207970 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 22:04:53.280679 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.280653 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6d7a08-3da9-4f23-80a6-faba18eb394d-kube-api-access-clsf8\" (UniqueName: \"kubernetes.io/projected/7f6d7a08-3da9-4f23-80a6-faba18eb394d-kube-api-access-clsf8\") pod \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " Nov 02 22:04:53.280807 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.280688 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-dshm\") pod \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " Nov 02 22:04:53.280807 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.280747 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tls-certs\") pod \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " Nov 02 22:04:53.280807 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.280768 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-model-cache\") pod \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " Nov 02 22:04:53.280931 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.280833 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-home\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-home\") pod \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " Nov 02 22:04:53.281142 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.281098 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-model-cache" pod "7f6d7a08-3da9-4f23-80a6-faba18eb394d" (UID: "7f6d7a08-3da9-4f23-80a6-faba18eb394d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.281263 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.281142 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-home" pod "7f6d7a08-3da9-4f23-80a6-faba18eb394d" (UID: "7f6d7a08-3da9-4f23-80a6-faba18eb394d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.281329 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.281276 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.281329 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.281297 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.282911 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.282886 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-dshm" pod "7f6d7a08-3da9-4f23-80a6-faba18eb394d" (UID: "7f6d7a08-3da9-4f23-80a6-faba18eb394d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.283022 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.282891 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tls-certs" pod "7f6d7a08-3da9-4f23-80a6-faba18eb394d" (UID: "7f6d7a08-3da9-4f23-80a6-faba18eb394d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:53.283022 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.282998 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6d7a08-3da9-4f23-80a6-faba18eb394d-kube-api-access-clsf8" pod "7f6d7a08-3da9-4f23-80a6-faba18eb394d" (UID: "7f6d7a08-3da9-4f23-80a6-faba18eb394d"). InnerVolumeSpecName "kube-api-access-clsf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:53.382028 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.381992 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tmp-dir\") pod \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\" (UID: \"7f6d7a08-3da9-4f23-80a6-faba18eb394d\") " Nov 02 22:04:53.382277 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.382254 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tmp-dir" pod "7f6d7a08-3da9-4f23-80a6-faba18eb394d" (UID: "7f6d7a08-3da9-4f23-80a6-faba18eb394d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:04:53.382561 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.382542 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.382612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.382571 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clsf8\" (UniqueName: \"kubernetes.io/projected/7f6d7a08-3da9-4f23-80a6-faba18eb394d-kube-api-access-clsf8\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.382612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.382585 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6d7a08-3da9-4f23-80a6-faba18eb394d-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.382612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.382601 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6d7a08-3da9-4f23-80a6-faba18eb394d-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.721080 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.721050 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4"] Nov 02 22:04:53.721277 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.721246 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" podUID="9c0380e3-4e17-4b1e-aa4d-42d596aa8425" containerName="maas-api" containerID="cri-o://6802fc78f1084634a5a0b1bbea7d3d003431624a79586b0ba6d3c0072b4e13b9" gracePeriod=30 Nov 02 22:04:53.930550 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.930497 4510 generic.go:356] "Generic (PLEG): container finished" podID="9c0380e3-4e17-4b1e-aa4d-42d596aa8425" containerID="6802fc78f1084634a5a0b1bbea7d3d003431624a79586b0ba6d3c0072b4e13b9" exitCode=0 Nov 02 22:04:53.930794 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.930555 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" event={"ID":"9c0380e3-4e17-4b1e-aa4d-42d596aa8425","Type":"ContainerDied","Data":"6802fc78f1084634a5a0b1bbea7d3d003431624a79586b0ba6d3c0072b4e13b9"} Nov 02 22:04:53.932324 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.932303 4510 generic.go:356] "Generic (PLEG): container finished" podID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerID="85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6" exitCode=0 Nov 02 22:04:53.932560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.932340 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" event={"ID":"7f6d7a08-3da9-4f23-80a6-faba18eb394d","Type":"ContainerDied","Data":"85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6"} Nov 02 22:04:53.932560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.932371 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" Nov 02 22:04:53.932560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.932374 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" event={"ID":"7f6d7a08-3da9-4f23-80a6-faba18eb394d","Type":"ContainerDied","Data":"c51e2c261d4f749167dcf554c3db0f4772fcc625e13a2dda4563cecbf122f1ec"} Nov 02 22:04:53.932560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.932429 4510 scope.go:122] "RemoveContainer" containerID="85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6" Nov 02 22:04:53.943662 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.943643 4510 scope.go:122] "RemoveContainer" containerID="85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6" Nov 02 22:04:53.943972 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:04:53.943954 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6\": container with ID starting with 85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6 not found: ID does not exist" containerID="85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6" Nov 02 22:04:53.944025 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.943981 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6"} err="failed to get container status \"85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6\": rpc error: code = NotFound desc = could not find container \"85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6\": container with ID starting with 85503c788b665d3d5f568cda801d4a77a43b7325a7334bccd1344048bab2afc6 not found: ID does not exist" Nov 02 22:04:53.982653 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.982592 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 22:04:53.990123 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.990094 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-metrics-certs\") pod \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " Nov 02 22:04:53.990243 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.990137 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-openshift-service-ca\") pod \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " Nov 02 22:04:53.990314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.990278 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls\") pod \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " Nov 02 22:04:53.990373 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.990324 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-kube-api-access-6296d\" (UniqueName: \"kubernetes.io/projected/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-kube-api-access-6296d\") pod \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\" (UID: \"9c0380e3-4e17-4b1e-aa4d-42d596aa8425\") " Nov 02 22:04:53.990558 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.990533 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-openshift-service-ca" pod "9c0380e3-4e17-4b1e-aa4d-42d596aa8425" (UID: "9c0380e3-4e17-4b1e-aa4d-42d596aa8425"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:04:53.990740 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.990719 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:53.992106 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.992086 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-metrics-certs" pod "9c0380e3-4e17-4b1e-aa4d-42d596aa8425" (UID: "9c0380e3-4e17-4b1e-aa4d-42d596aa8425"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:53.992284 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.992268 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls" pod "9c0380e3-4e17-4b1e-aa4d-42d596aa8425" (UID: "9c0380e3-4e17-4b1e-aa4d-42d596aa8425"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:04:53.992329 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:53.992295 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-kube-api-access-6296d" pod "9c0380e3-4e17-4b1e-aa4d-42d596aa8425" (UID: "9c0380e3-4e17-4b1e-aa4d-42d596aa8425"). InnerVolumeSpecName "kube-api-access-6296d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:04:54.086188 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.086131 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj"] Nov 02 22:04:54.090291 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.090225 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj"] Nov 02 22:04:54.091844 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.091811 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:54.091844 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.091843 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6296d\" (UniqueName: \"kubernetes.io/projected/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-kube-api-access-6296d\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:54.092017 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.091860 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c0380e3-4e17-4b1e-aa4d-42d596aa8425-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:04:54.120629 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.120594 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4e2b27/distinct-2-w4-4e2b27-kserve-94b5c699d-fpfcj" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:04:54.152857 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.152832 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" path="/var/lib/kubelet/pods/7f6d7a08-3da9-4f23-80a6-faba18eb394d/volumes" Nov 02 22:04:54.939029 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.938995 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" Nov 02 22:04:54.939029 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.939010 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4" event={"ID":"9c0380e3-4e17-4b1e-aa4d-42d596aa8425","Type":"ContainerDied","Data":"7459269d7ac978fb1c10250ba014663ade8604783408bda043d9eab85d5571cb"} Nov 02 22:04:54.939263 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:54.939109 4510 scope.go:122] "RemoveContainer" containerID="6802fc78f1084634a5a0b1bbea7d3d003431624a79586b0ba6d3c0072b4e13b9" Nov 02 22:04:55.091081 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:55.091051 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4"] Nov 02 22:04:55.137087 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:55.137056 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9a6cf2-67fcb949dc-wtdk4"] Nov 02 22:04:56.152478 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:04:56.152442 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c0380e3-4e17-4b1e-aa4d-42d596aa8425" path="/var/lib/kubelet/pods/9c0380e3-4e17-4b1e-aa4d-42d596aa8425/volumes" Nov 02 22:05:02.360492 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:02.360452 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5"] Nov 02 22:05:02.360962 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:02.360685 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" containerID="cri-o://e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744" gracePeriod=30 Nov 02 22:05:03.712572 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:03.712532 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:05:04.356191 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.356159 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7"] Nov 02 22:05:04.356384 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.356366 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" podUID="2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" containerName="maas-api" containerID="cri-o://f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a" gracePeriod=30 Nov 02 22:05:04.599915 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.599889 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 22:05:04.697633 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.697601 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls\") pod \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " Nov 02 22:05:04.697633 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.697637 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-kube-api-access-rkrcq\" (UniqueName: \"kubernetes.io/projected/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-kube-api-access-rkrcq\") pod \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " Nov 02 22:05:04.697885 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.697737 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-metrics-certs\") pod \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " Nov 02 22:05:04.697885 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.697777 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-openshift-service-ca\") pod \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\" (UID: \"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0\") " Nov 02 22:05:04.698157 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.698129 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-openshift-service-ca" pod "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" (UID: "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:05:04.699871 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.699843 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-kube-api-access-rkrcq" pod "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" (UID: "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0"). InnerVolumeSpecName "kube-api-access-rkrcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:04.699968 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.699924 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls" pod "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" (UID: "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:04.699968 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.699945 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-metrics-certs" pod "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" (UID: "2c674c14-2c24-4aa7-88f7-f2d9d0364bf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:04.799138 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.799102 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:04.799138 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.799136 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:04.799548 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.799153 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrcq\" (UniqueName: \"kubernetes.io/projected/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-kube-api-access-rkrcq\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:04.799548 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.799168 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:04.997317 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.997228 4510 generic.go:356] "Generic (PLEG): container finished" podID="2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" containerID="f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a" exitCode=0 Nov 02 22:05:04.997317 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.997277 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" Nov 02 22:05:04.997583 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.997309 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" event={"ID":"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0","Type":"ContainerDied","Data":"f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a"} Nov 02 22:05:04.997583 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.997367 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7" event={"ID":"2c674c14-2c24-4aa7-88f7-f2d9d0364bf0","Type":"ContainerDied","Data":"fea10c5adbe2195ce1efeea1d3991d025135d6c21c40c1f59d1abdc5fe1d7416"} Nov 02 22:05:04.997583 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:04.997396 4510 scope.go:122] "RemoveContainer" containerID="f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a" Nov 02 22:05:05.008846 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:05.008823 4510 scope.go:122] "RemoveContainer" containerID="f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a" Nov 02 22:05:05.009108 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:05:05.009091 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a\": container with ID starting with f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a not found: ID does not exist" containerID="f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a" Nov 02 22:05:05.009161 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:05.009117 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a"} err="failed to get container status \"f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a\": rpc error: code = NotFound desc = could not find container \"f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a\": container with ID starting with f2f881f5d96d1018c132136880cbbdc34dee0c6be519d0e1f9cb7940d35a115a not found: ID does not exist" Nov 02 22:05:05.155427 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:05.155394 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7"] Nov 02 22:05:05.161085 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:05.161050 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4d9ea0-5ccc9586b4-x2pj7"] Nov 02 22:05:06.153413 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:06.153381 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" path="/var/lib/kubelet/pods/2c674c14-2c24-4aa7-88f7-f2d9d0364bf0/volumes" Nov 02 22:05:06.502893 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:06.502816 4510 patch_prober.go:29] interesting pod/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:05:06.502893 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:06.502871 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:05:07.608267 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.608240 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 22:05:07.730053 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.729959 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-certs\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730053 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730004 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-token\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-token\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730053 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730038 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-data\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730347 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730080 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-podinfo\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730347 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730232 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b023881e-83cb-44b7-9db4-246dd5665b6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b023881e-83cb-44b7-9db4-246dd5665b6a-istiod-ca-cert\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730347 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730273 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-certs" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:07.730347 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730330 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-socket\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730585 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730364 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-envoy\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730585 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730439 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-kube-api-access-nj6fm\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-kube-api-access-nj6fm\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730585 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730534 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-credential-socket\") pod \"b023881e-83cb-44b7-9db4-246dd5665b6a\" (UID: \"b023881e-83cb-44b7-9db4-246dd5665b6a\") " Nov 02 22:05:07.730759 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730441 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-data" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:07.730759 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730686 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-socket" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:07.730759 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730722 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b023881e-83cb-44b7-9db4-246dd5665b6a-istiod-ca-cert" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:05:07.730981 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.730951 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-credential-socket" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:07.731165 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.731003 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.731165 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.731021 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.731165 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.731034 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b023881e-83cb-44b7-9db4-246dd5665b6a-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.731165 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.731046 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.732915 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.732890 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-podinfo" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:05:07.733155 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.733130 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-token" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:07.733271 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.733169 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-kube-api-access-nj6fm" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "kube-api-access-nj6fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:07.733619 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.733598 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-envoy" pod "b023881e-83cb-44b7-9db4-246dd5665b6a" (UID: "b023881e-83cb-44b7-9db4-246dd5665b6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:05:07.832350 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.832315 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.832350 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.832347 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.832350 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.832357 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.832626 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.832366 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj6fm\" (UniqueName: \"kubernetes.io/projected/b023881e-83cb-44b7-9db4-246dd5665b6a-kube-api-access-nj6fm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:07.832626 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:07.832377 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b023881e-83cb-44b7-9db4-246dd5665b6a-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:08.016011 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.015924 4510 generic.go:356] "Generic (PLEG): container finished" podID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerID="e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744" exitCode=0 Nov 02 22:05:08.016011 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.015981 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" event={"ID":"b023881e-83cb-44b7-9db4-246dd5665b6a","Type":"ContainerDied","Data":"e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744"} Nov 02 22:05:08.016011 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.015987 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" Nov 02 22:05:08.016248 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.016035 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5" event={"ID":"b023881e-83cb-44b7-9db4-246dd5665b6a","Type":"ContainerDied","Data":"2479b626410d030d952dcd69c4cfaa03d866d08b16d841def3d2c22ce4209132"} Nov 02 22:05:08.016248 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.016067 4510 scope.go:122] "RemoveContainer" containerID="e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744" Nov 02 22:05:08.026058 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.026038 4510 scope.go:122] "RemoveContainer" containerID="2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3" Nov 02 22:05:08.033902 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.033883 4510 scope.go:122] "RemoveContainer" containerID="e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744" Nov 02 22:05:08.034163 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:05:08.034144 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744\": container with ID starting with e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744 not found: ID does not exist" containerID="e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744" Nov 02 22:05:08.034220 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.034172 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744"} err="failed to get container status \"e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744\": rpc error: code = NotFound desc = could not find container \"e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744\": container with ID starting with e791c07584c81cca57052cab8acbb26affc1a5a2603ac495dd1742db9ef68744 not found: ID does not exist" Nov 02 22:05:08.034220 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.034189 4510 scope.go:122] "RemoveContainer" containerID="2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3" Nov 02 22:05:08.034397 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:05:08.034381 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3\": container with ID starting with 2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3 not found: ID does not exist" containerID="2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3" Nov 02 22:05:08.034437 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.034401 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3"} err="failed to get container status \"2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3\": rpc error: code = NotFound desc = could not find container \"2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3\": container with ID starting with 2241d1be4f4ab4848ac3135351d715cee4712578ed41c51e902131522e4ac1d3 not found: ID does not exist" Nov 02 22:05:08.168846 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.168820 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5"] Nov 02 22:05:08.172167 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:08.172141 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9a6cf2-openshift-default-5d598fdc7c-gnxj5"] Nov 02 22:05:10.152190 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:10.152161 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" path="/var/lib/kubelet/pods/b023881e-83cb-44b7-9db4-246dd5665b6a/volumes" Nov 02 22:05:15.008570 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.008538 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r"] Nov 02 22:05:15.009083 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.008754 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" podUID="5bddd2c8-5804-4ae2-a11d-c708cbc56e59" containerName="maas-api" containerID="cri-o://4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd" gracePeriod=30 Nov 02 22:05:15.270730 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.270670 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 22:05:15.410560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.410531 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-kube-api-access-fr6mv\" (UniqueName: \"kubernetes.io/projected/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-kube-api-access-fr6mv\") pod \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " Nov 02 22:05:15.410760 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.410591 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-openshift-service-ca\") pod \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " Nov 02 22:05:15.410760 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.410698 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls\") pod \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " Nov 02 22:05:15.410760 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.410736 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-metrics-certs\") pod \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\" (UID: \"5bddd2c8-5804-4ae2-a11d-c708cbc56e59\") " Nov 02 22:05:15.410971 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.410948 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-openshift-service-ca" pod "5bddd2c8-5804-4ae2-a11d-c708cbc56e59" (UID: "5bddd2c8-5804-4ae2-a11d-c708cbc56e59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:05:15.411169 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.411154 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:15.412828 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.412803 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-metrics-certs" pod "5bddd2c8-5804-4ae2-a11d-c708cbc56e59" (UID: "5bddd2c8-5804-4ae2-a11d-c708cbc56e59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:15.413187 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.413163 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls" pod "5bddd2c8-5804-4ae2-a11d-c708cbc56e59" (UID: "5bddd2c8-5804-4ae2-a11d-c708cbc56e59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:05:15.413270 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.413202 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-kube-api-access-fr6mv" pod "5bddd2c8-5804-4ae2-a11d-c708cbc56e59" (UID: "5bddd2c8-5804-4ae2-a11d-c708cbc56e59"). InnerVolumeSpecName "kube-api-access-fr6mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:05:15.512298 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.512269 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6mv\" (UniqueName: \"kubernetes.io/projected/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-kube-api-access-fr6mv\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:15.512298 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.512296 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:15.512298 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:15.512306 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bddd2c8-5804-4ae2-a11d-c708cbc56e59-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:05:15.530515 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:05:15.530443 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:05:16.060016 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.059979 4510 generic.go:356] "Generic (PLEG): container finished" podID="5bddd2c8-5804-4ae2-a11d-c708cbc56e59" containerID="4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd" exitCode=0 Nov 02 22:05:16.060464 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.060033 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" Nov 02 22:05:16.060464 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.060061 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" event={"ID":"5bddd2c8-5804-4ae2-a11d-c708cbc56e59","Type":"ContainerDied","Data":"4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd"} Nov 02 22:05:16.060464 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.060113 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r" event={"ID":"5bddd2c8-5804-4ae2-a11d-c708cbc56e59","Type":"ContainerDied","Data":"344484c5cb37d4715accce8f5791baf846362ac3d24a56933ddc3ae52946d795"} Nov 02 22:05:16.060464 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.060136 4510 scope.go:122] "RemoveContainer" containerID="4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd" Nov 02 22:05:16.073314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.073289 4510 scope.go:122] "RemoveContainer" containerID="4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd" Nov 02 22:05:16.073592 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:05:16.073575 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd\": container with ID starting with 4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd not found: ID does not exist" containerID="4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd" Nov 02 22:05:16.073649 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.073600 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd"} err="failed to get container status \"4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd\": rpc error: code = NotFound desc = could not find container \"4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd\": container with ID starting with 4546049c6f2b2dacda77679510be20e3fdeb757fabe06abcb32a0d783ba2e8bd not found: ID does not exist" Nov 02 22:05:16.211735 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.211702 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r"] Nov 02 22:05:16.257963 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:16.257937 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4e2b27-7b959f4485-bql4r"] Nov 02 22:05:18.152704 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:18.152673 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bddd2c8-5804-4ae2-a11d-c708cbc56e59" path="/var/lib/kubelet/pods/5bddd2c8-5804-4ae2-a11d-c708cbc56e59/volumes" Nov 02 22:05:37.251948 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:05:37.251913 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:05:45.533003 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:05:45.532966 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:06:08.416932 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:06:08.416854 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:06:15.530180 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:06:15.530133 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:06:45.533838 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:06:45.533797 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:06:49.080188 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:06:49.080152 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:07:15.530104 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:07:15.530065 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:07:45.531127 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:07:45.531047 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:08:15.530276 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:08:15.530237 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:08:23.937696 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:23.937665 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj"] Nov 02 22:08:28.769962 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:28.769930 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:08:38.957447 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:38.957412 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" podUID="42d51387-efaa-4d06-b813-25b70ebbf594" containerName="main" containerID="cri-o://9b91dfd535e2708e1099733d4cca14bb27cdf986008e805a35362c735c1901fb" gracePeriod=45 Nov 02 22:08:39.086796 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.086764 4510 generic.go:356] "Generic (PLEG): container finished" podID="42d51387-efaa-4d06-b813-25b70ebbf594" containerID="9b91dfd535e2708e1099733d4cca14bb27cdf986008e805a35362c735c1901fb" exitCode=0 Nov 02 22:08:39.086955 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.086836 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" event={"ID":"42d51387-efaa-4d06-b813-25b70ebbf594","Type":"ContainerDied","Data":"9b91dfd535e2708e1099733d4cca14bb27cdf986008e805a35362c735c1901fb"} Nov 02 22:08:39.210480 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.210422 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:08:39.288871 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.288840 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-tmp-dir\") pod \"42d51387-efaa-4d06-b813-25b70ebbf594\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " Nov 02 22:08:39.289026 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.288879 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-home\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-home\") pod \"42d51387-efaa-4d06-b813-25b70ebbf594\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " Nov 02 22:08:39.289026 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.288984 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d51387-efaa-4d06-b813-25b70ebbf594-kube-api-access-47nr9\" (UniqueName: \"kubernetes.io/projected/42d51387-efaa-4d06-b813-25b70ebbf594-kube-api-access-47nr9\") pod \"42d51387-efaa-4d06-b813-25b70ebbf594\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " Nov 02 22:08:39.289026 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.289010 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-model-cache\") pod \"42d51387-efaa-4d06-b813-25b70ebbf594\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " Nov 02 22:08:39.289207 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.289036 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-dshm\") pod \"42d51387-efaa-4d06-b813-25b70ebbf594\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " Nov 02 22:08:39.289207 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.289091 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42d51387-efaa-4d06-b813-25b70ebbf594-tls-certs\" (UniqueName: \"kubernetes.io/secret/42d51387-efaa-4d06-b813-25b70ebbf594-tls-certs\") pod \"42d51387-efaa-4d06-b813-25b70ebbf594\" (UID: \"42d51387-efaa-4d06-b813-25b70ebbf594\") " Nov 02 22:08:39.289207 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.289116 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-tmp-dir" pod "42d51387-efaa-4d06-b813-25b70ebbf594" (UID: "42d51387-efaa-4d06-b813-25b70ebbf594"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:39.289345 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.289204 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-home" pod "42d51387-efaa-4d06-b813-25b70ebbf594" (UID: "42d51387-efaa-4d06-b813-25b70ebbf594"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:39.289345 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.289264 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-model-cache" pod "42d51387-efaa-4d06-b813-25b70ebbf594" (UID: "42d51387-efaa-4d06-b813-25b70ebbf594"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:39.290216 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.290195 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:39.290403 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.290380 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:39.290550 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.290405 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:39.291493 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.291471 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-dshm" pod "42d51387-efaa-4d06-b813-25b70ebbf594" (UID: "42d51387-efaa-4d06-b813-25b70ebbf594"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:08:39.291642 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.291542 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d51387-efaa-4d06-b813-25b70ebbf594-kube-api-access-47nr9" pod "42d51387-efaa-4d06-b813-25b70ebbf594" (UID: "42d51387-efaa-4d06-b813-25b70ebbf594"). InnerVolumeSpecName "kube-api-access-47nr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:39.291872 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.291853 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d51387-efaa-4d06-b813-25b70ebbf594-tls-certs" pod "42d51387-efaa-4d06-b813-25b70ebbf594" (UID: "42d51387-efaa-4d06-b813-25b70ebbf594"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:39.392023 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.391990 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d51387-efaa-4d06-b813-25b70ebbf594-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:39.392023 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.392022 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47nr9\" (UniqueName: \"kubernetes.io/projected/42d51387-efaa-4d06-b813-25b70ebbf594-kube-api-access-47nr9\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:39.392205 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:39.392032 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d51387-efaa-4d06-b813-25b70ebbf594-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:40.093406 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:40.093379 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" Nov 02 22:08:40.093872 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:40.093423 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj" event={"ID":"42d51387-efaa-4d06-b813-25b70ebbf594","Type":"ContainerDied","Data":"6dfb58b4a91e83a41bbfc5dc753e91a1e76d5628cd3cfce3d56b959cdb795cd3"} Nov 02 22:08:40.093872 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:40.093476 4510 scope.go:122] "RemoveContainer" containerID="9b91dfd535e2708e1099733d4cca14bb27cdf986008e805a35362c735c1901fb" Nov 02 22:08:40.244922 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:40.244894 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj"] Nov 02 22:08:40.248994 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:40.248966 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c4a6b9/test-model-c4a6b9-kserve-74bd4b85-qnkbj"] Nov 02 22:08:42.153430 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:42.153399 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d51387-efaa-4d06-b813-25b70ebbf594" path="/var/lib/kubelet/pods/42d51387-efaa-4d06-b813-25b70ebbf594/volumes" Nov 02 22:08:45.531806 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:08:45.531764 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:08:47.214768 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:08:47.214740 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:09:01.999671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:01.999636 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:09:11.743806 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:11.743720 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v"] Nov 02 22:09:11.744255 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:11.743973 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" podUID="93ae6126-729e-4525-8e03-d9cda4256a3f" containerName="maas-api" containerID="cri-o://edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7" gracePeriod=30 Nov 02 22:09:11.997620 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:11.997565 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:09:12.102580 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.102468 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-metrics-certs\") pod \"93ae6126-729e-4525-8e03-d9cda4256a3f\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " Nov 02 22:09:12.102580 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.102500 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ae6126-729e-4525-8e03-d9cda4256a3f-kube-api-access-xc4xc\" (UniqueName: \"kubernetes.io/projected/93ae6126-729e-4525-8e03-d9cda4256a3f-kube-api-access-xc4xc\") pod \"93ae6126-729e-4525-8e03-d9cda4256a3f\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " Nov 02 22:09:12.102580 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.102550 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-maas-api-tls\") pod \"93ae6126-729e-4525-8e03-d9cda4256a3f\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " Nov 02 22:09:12.102850 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.102694 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ae6126-729e-4525-8e03-d9cda4256a3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae6126-729e-4525-8e03-d9cda4256a3f-openshift-service-ca\") pod \"93ae6126-729e-4525-8e03-d9cda4256a3f\" (UID: \"93ae6126-729e-4525-8e03-d9cda4256a3f\") " Nov 02 22:09:12.103148 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.103122 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ae6126-729e-4525-8e03-d9cda4256a3f-openshift-service-ca" pod "93ae6126-729e-4525-8e03-d9cda4256a3f" (UID: "93ae6126-729e-4525-8e03-d9cda4256a3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:12.104808 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.104782 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-metrics-certs" pod "93ae6126-729e-4525-8e03-d9cda4256a3f" (UID: "93ae6126-729e-4525-8e03-d9cda4256a3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:12.105152 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.105120 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ae6126-729e-4525-8e03-d9cda4256a3f-kube-api-access-xc4xc" pod "93ae6126-729e-4525-8e03-d9cda4256a3f" (UID: "93ae6126-729e-4525-8e03-d9cda4256a3f"). InnerVolumeSpecName "kube-api-access-xc4xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:12.105261 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.105168 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-maas-api-tls" pod "93ae6126-729e-4525-8e03-d9cda4256a3f" (UID: "93ae6126-729e-4525-8e03-d9cda4256a3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:12.203862 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.203834 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93ae6126-729e-4525-8e03-d9cda4256a3f-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:12.203862 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.203860 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:12.204018 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.203869 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc4xc\" (UniqueName: \"kubernetes.io/projected/93ae6126-729e-4525-8e03-d9cda4256a3f-kube-api-access-xc4xc\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:12.204018 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.203878 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93ae6126-729e-4525-8e03-d9cda4256a3f-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:12.249255 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.249180 4510 generic.go:356] "Generic (PLEG): container finished" podID="93ae6126-729e-4525-8e03-d9cda4256a3f" containerID="edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7" exitCode=0 Nov 02 22:09:12.249255 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.249233 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" Nov 02 22:09:12.249459 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.249272 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" event={"ID":"93ae6126-729e-4525-8e03-d9cda4256a3f","Type":"ContainerDied","Data":"edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7"} Nov 02 22:09:12.249459 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.249326 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v" event={"ID":"93ae6126-729e-4525-8e03-d9cda4256a3f","Type":"ContainerDied","Data":"7e8018f25fae2e9c0363c1c38c12f4c56ce81fa1525fc0a27868bf56de10ce28"} Nov 02 22:09:12.249459 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.249348 4510 scope.go:122] "RemoveContainer" containerID="edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7" Nov 02 22:09:12.258342 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.258328 4510 scope.go:122] "RemoveContainer" containerID="edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7" Nov 02 22:09:12.258631 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:09:12.258611 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7\": container with ID starting with edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7 not found: ID does not exist" containerID="edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7" Nov 02 22:09:12.258702 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.258637 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7"} err="failed to get container status \"edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7\": rpc error: code = NotFound desc = could not find container \"edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7\": container with ID starting with edab2a789d863bcf401d1b80e27fb5b471b19c79fa3aa1ea33beeb8f261754c7 not found: ID does not exist" Nov 02 22:09:12.400328 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.400303 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v"] Nov 02 22:09:12.402009 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:12.401990 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4a6b9-58cc6868d9-vmm6v"] Nov 02 22:09:14.158950 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:14.158908 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ae6126-729e-4525-8e03-d9cda4256a3f" path="/var/lib/kubelet/pods/93ae6126-729e-4525-8e03-d9cda4256a3f/volumes" Nov 02 22:09:15.530079 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:09:15.530041 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:09:17.881269 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:17.881236 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:09:40.573372 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.573336 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx"] Nov 02 22:09:40.574014 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.573597 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" podUID="5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" containerName="maas-api" containerID="cri-o://d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043" gracePeriod=30 Nov 02 22:09:40.836183 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.836121 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:09:40.893592 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.893562 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-metrics-certs\") pod \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " Nov 02 22:09:40.893763 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.893618 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-openshift-service-ca\") pod \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " Nov 02 22:09:40.893763 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.893735 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-kube-api-access-9s78c\" (UniqueName: \"kubernetes.io/projected/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-kube-api-access-9s78c\") pod \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " Nov 02 22:09:40.893903 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.893875 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls\") pod \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\" (UID: \"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed\") " Nov 02 22:09:40.894005 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.893985 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-openshift-service-ca" pod "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" (UID: "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:40.894196 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.894176 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:40.895876 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.895849 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-kube-api-access-9s78c" pod "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" (UID: "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed"). InnerVolumeSpecName "kube-api-access-9s78c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:40.895876 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.895856 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls" pod "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" (UID: "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:40.896013 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.895923 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-metrics-certs" pod "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" (UID: "5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:40.952971 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.952937 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg"] Nov 02 22:09:40.953206 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.953165 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" containerID="cri-o://a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff" gracePeriod=30 Nov 02 22:09:40.995657 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.995629 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:40.995657 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.995653 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:40.995835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:40.995665 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s78c\" (UniqueName: \"kubernetes.io/projected/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed-kube-api-access-9s78c\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:41.387913 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.387879 4510 generic.go:356] "Generic (PLEG): container finished" podID="5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" containerID="d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043" exitCode=0 Nov 02 22:09:41.388119 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.387926 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" Nov 02 22:09:41.388119 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.387967 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" event={"ID":"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed","Type":"ContainerDied","Data":"d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043"} Nov 02 22:09:41.388119 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.388025 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx" event={"ID":"5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed","Type":"ContainerDied","Data":"fcca36abf1353781bd8cc3151936fb520d92cf8dd77db866f136ae330351d592"} Nov 02 22:09:41.388119 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.388054 4510 scope.go:122] "RemoveContainer" containerID="d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043" Nov 02 22:09:41.398252 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.398236 4510 scope.go:122] "RemoveContainer" containerID="d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043" Nov 02 22:09:41.398502 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:09:41.398484 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043\": container with ID starting with d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043 not found: ID does not exist" containerID="d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043" Nov 02 22:09:41.398583 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.398507 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043"} err="failed to get container status \"d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043\": rpc error: code = NotFound desc = could not find container \"d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043\": container with ID starting with d094dc514b484cc5e3a5d6c01fea91510ba0fb9651a0e7eb2d4c798729936043 not found: ID does not exist" Nov 02 22:09:41.539448 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.539416 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx"] Nov 02 22:09:41.542436 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:41.542409 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-49c06d-66c9469bd9-zkmpx"] Nov 02 22:09:42.153620 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:42.153590 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" path="/var/lib/kubelet/pods/5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed/volumes" Nov 02 22:09:45.532805 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:09:45.532759 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:09:46.205701 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.205675 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 22:09:46.212600 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.212578 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:09:46.250054 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.249997 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-credential-socket\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250054 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250027 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd937071-bddd-46ae-b7a6-1a8523442445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd937071-bddd-46ae-b7a6-1a8523442445-istio-podinfo\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250054 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250046 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-envoy\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250284 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250107 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd937071-bddd-46ae-b7a6-1a8523442445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd937071-bddd-46ae-b7a6-1a8523442445-istiod-ca-cert\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250284 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250144 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-kube-api-access-b9xpr\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-kube-api-access-b9xpr\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250284 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250177 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-certs\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250284 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250252 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-data\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250284 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250262 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-credential-socket" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:46.250601 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250356 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-socket\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250601 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250404 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-istio-token\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-istio-token\") pod \"cd937071-bddd-46ae-b7a6-1a8523442445\" (UID: \"cd937071-bddd-46ae-b7a6-1a8523442445\") " Nov 02 22:09:46.250601 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250467 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-certs" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:46.250817 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250591 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-data" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:46.250817 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250639 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd937071-bddd-46ae-b7a6-1a8523442445-istiod-ca-cert" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:46.250817 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250711 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-socket" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:46.250981 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250882 4510 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-data\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.250981 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250901 4510 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.250981 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250916 4510 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-credential-socket\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.250981 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250926 4510 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd937071-bddd-46ae-b7a6-1a8523442445-istiod-ca-cert\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.250981 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.250936 4510 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-workload-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.252449 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.252426 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-envoy" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:09:46.252644 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.252615 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-kube-api-access-b9xpr" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "kube-api-access-b9xpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:46.252758 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.252739 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd937071-bddd-46ae-b7a6-1a8523442445-istio-podinfo" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:09:46.252899 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.252879 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-istio-token" pod "cd937071-bddd-46ae-b7a6-1a8523442445" (UID: "cd937071-bddd-46ae-b7a6-1a8523442445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:46.351827 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.351800 4510 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-istio-token\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.351827 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.351826 4510 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd937071-bddd-46ae-b7a6-1a8523442445-istio-podinfo\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.352017 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.351835 4510 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd937071-bddd-46ae-b7a6-1a8523442445-istio-envoy\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.352017 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.351846 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xpr\" (UniqueName: \"kubernetes.io/projected/cd937071-bddd-46ae-b7a6-1a8523442445-kube-api-access-b9xpr\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:46.414834 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.414795 4510 generic.go:356] "Generic (PLEG): container finished" podID="cd937071-bddd-46ae-b7a6-1a8523442445" containerID="a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff" exitCode=0 Nov 02 22:09:46.414975 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.414855 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" Nov 02 22:09:46.414975 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.414886 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" event={"ID":"cd937071-bddd-46ae-b7a6-1a8523442445","Type":"ContainerDied","Data":"a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff"} Nov 02 22:09:46.414975 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.414945 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg" event={"ID":"cd937071-bddd-46ae-b7a6-1a8523442445","Type":"ContainerDied","Data":"87e33a4fa0cb3eedd3f5ccdb5b3cd2621deccea0c8b896bf5ab20dfaa1b70017"} Nov 02 22:09:46.414975 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.414974 4510 scope.go:122] "RemoveContainer" containerID="a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff" Nov 02 22:09:46.424750 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.424727 4510 scope.go:122] "RemoveContainer" containerID="a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff" Nov 02 22:09:46.425055 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:09:46.425030 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff\": container with ID starting with a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff not found: ID does not exist" containerID="a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff" Nov 02 22:09:46.425150 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.425069 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff"} err="failed to get container status \"a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff\": rpc error: code = NotFound desc = could not find container \"a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff\": container with ID starting with a2e30e6a42c2dcd691e5dc71353e275f676272bd075636d606ba4c984bcc98ff not found: ID does not exist" Nov 02 22:09:46.566888 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.566811 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg"] Nov 02 22:09:46.576459 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:46.576425 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sz8lg"] Nov 02 22:09:48.152565 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:48.152537 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" path="/var/lib/kubelet/pods/cd937071-bddd-46ae-b7a6-1a8523442445/volumes" Nov 02 22:09:50.253748 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.253712 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj"] Nov 02 22:09:50.254206 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254189 4510 cpu_manager.go:414] "containerMap: removing container" podUID="93ae6126-729e-4525-8e03-d9cda4256a3f" containerName="maas-api" Nov 02 22:09:50.254254 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254207 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ae6126-729e-4525-8e03-d9cda4256a3f" containerName="maas-api" Nov 02 22:09:50.254254 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254223 4510 cpu_manager.go:414] "containerMap: removing container" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" Nov 02 22:09:50.254254 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254227 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" Nov 02 22:09:50.254254 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254243 4510 cpu_manager.go:414] "containerMap: removing container" podUID="5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" containerName="maas-api" Nov 02 22:09:50.254254 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254249 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" containerName="maas-api" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254257 4510 cpu_manager.go:414] "containerMap: removing container" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254262 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254271 4510 cpu_manager.go:414] "containerMap: removing container" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254275 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254286 4510 cpu_manager.go:414] "containerMap: removing container" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254290 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254297 4510 cpu_manager.go:414] "containerMap: removing container" podUID="9c0380e3-4e17-4b1e-aa4d-42d596aa8425" containerName="maas-api" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254301 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c0380e3-4e17-4b1e-aa4d-42d596aa8425" containerName="maas-api" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254311 4510 cpu_manager.go:414] "containerMap: removing container" podUID="42d51387-efaa-4d06-b813-25b70ebbf594" containerName="main" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254316 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d51387-efaa-4d06-b813-25b70ebbf594" containerName="main" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254323 4510 cpu_manager.go:414] "containerMap: removing container" podUID="5bddd2c8-5804-4ae2-a11d-c708cbc56e59" containerName="maas-api" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254327 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bddd2c8-5804-4ae2-a11d-c708cbc56e59" containerName="maas-api" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254336 4510 cpu_manager.go:414] "containerMap: removing container" podUID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" containerName="payload-processing" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254340 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" containerName="payload-processing" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254351 4510 cpu_manager.go:414] "containerMap: removing container" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254355 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254362 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254366 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254373 4510 cpu_manager.go:414] "containerMap: removing container" podUID="2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" containerName="maas-api" Nov 02 22:09:50.254426 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254377 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" containerName="maas-api" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254437 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ae6126-729e-4525-8e03-d9cda4256a3f" containerName="maas-api" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254446 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="baea62ff-7e44-4e92-83b4-018d05ecc226" containerName="main" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254454 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7f1dfb-0d83-4ecd-ab73-3b01b24fc6ed" containerName="maas-api" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254460 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254466 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd937071-bddd-46ae-b7a6-1a8523442445" containerName="istio-proxy" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254472 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b023881e-83cb-44b7-9db4-246dd5665b6a" containerName="istio-proxy" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254478 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c0380e3-4e17-4b1e-aa4d-42d596aa8425" containerName="maas-api" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254483 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d51387-efaa-4d06-b813-25b70ebbf594" containerName="main" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254488 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bddd2c8-5804-4ae2-a11d-c708cbc56e59" containerName="maas-api" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254494 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b30ae4c-6c6e-4975-9c84-6d76ba5c5544" containerName="payload-processing" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254499 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85f72e4-bab3-4754-8f49-463cc0c4daa9" containerName="main" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254504 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6d7a08-3da9-4f23-80a6-faba18eb394d" containerName="main" Nov 02 22:09:50.255103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.254508 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c674c14-2c24-4aa7-88f7-f2d9d0364bf0" containerName="maas-api" Nov 02 22:09:50.259317 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.259298 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.354081 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.354048 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:09:50.354219 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.354107 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:09:50.356124 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.356105 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj"] Nov 02 22:09:50.394087 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.394057 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.394087 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.394090 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ph9n\" (UniqueName: \"kubernetes.io/projected/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-kube-api-access-7ph9n\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.394266 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.394163 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.495590 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.495564 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.495720 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.495598 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ph9n\" (UniqueName: \"kubernetes.io/projected/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-kube-api-access-7ph9n\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.495720 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.495650 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.496292 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.496273 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.498030 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.498010 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.548104 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.548045 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ph9n\" (UniqueName: \"kubernetes.io/projected/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-kube-api-access-7ph9n\") pod \"maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.571929 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.571911 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:50.940079 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:50.940041 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj"] Nov 02 22:09:50.948094 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:09:50.948065 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7946c1ae_ab06_4d3c_b02a_a6534bb8a174.slice/crio-6f80d41981a8e1cf0bb55136cdb34d5af77287027f1c123fb41ebbe1a8ea0236 WatchSource:0}: Error finding container 6f80d41981a8e1cf0bb55136cdb34d5af77287027f1c123fb41ebbe1a8ea0236: Status 404 returned error can't find the container with id 6f80d41981a8e1cf0bb55136cdb34d5af77287027f1c123fb41ebbe1a8ea0236 Nov 02 22:09:51.440706 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:51.440671 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" event={"ID":"7946c1ae-ab06-4d3c-b02a-a6534bb8a174","Type":"ContainerStarted","Data":"6a59f75e0c82a6bcaee50d19b547a201475605e2ad7185c3daff0b0c82ce79ee"} Nov 02 22:09:51.441155 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:51.440747 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" event={"ID":"7946c1ae-ab06-4d3c-b02a-a6534bb8a174","Type":"ContainerStarted","Data":"6f80d41981a8e1cf0bb55136cdb34d5af77287027f1c123fb41ebbe1a8ea0236"} Nov 02 22:09:51.546904 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:51.546797 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" podStartSLOduration=1.5467834489999999 podStartE2EDuration="1.546783449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:51.544241362 +0000 UTC m=+3417.520539137" watchObservedRunningTime="2026-11-02 22:09:51.546783449 +0000 UTC m=+3417.523081223" Nov 02 22:09:52.448072 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:52.448034 4510 generic.go:356] "Generic (PLEG): container finished" podID="7946c1ae-ab06-4d3c-b02a-a6534bb8a174" containerID="6a59f75e0c82a6bcaee50d19b547a201475605e2ad7185c3daff0b0c82ce79ee" exitCode=0 Nov 02 22:09:52.448451 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:52.448103 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" event={"ID":"7946c1ae-ab06-4d3c-b02a-a6534bb8a174","Type":"ContainerDied","Data":"6a59f75e0c82a6bcaee50d19b547a201475605e2ad7185c3daff0b0c82ce79ee"} Nov 02 22:09:53.586691 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.586667 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:53.726060 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.725983 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-xks-ca\") pod \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " Nov 02 22:09:53.726220 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.726078 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-openshift-ca\") pod \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " Nov 02 22:09:53.726220 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.726154 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-kube-api-access-7ph9n\" (UniqueName: \"kubernetes.io/projected/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-kube-api-access-7ph9n\") pod \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\" (UID: \"7946c1ae-ab06-4d3c-b02a-a6534bb8a174\") " Nov 02 22:09:53.726467 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.726442 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-openshift-ca" pod "7946c1ae-ab06-4d3c-b02a-a6534bb8a174" (UID: "7946c1ae-ab06-4d3c-b02a-a6534bb8a174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:09:53.728040 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.728017 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-xks-ca" pod "7946c1ae-ab06-4d3c-b02a-a6534bb8a174" (UID: "7946c1ae-ab06-4d3c-b02a-a6534bb8a174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:09:53.728208 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.728190 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-kube-api-access-7ph9n" pod "7946c1ae-ab06-4d3c-b02a-a6534bb8a174" (UID: "7946c1ae-ab06-4d3c-b02a-a6534bb8a174"). InnerVolumeSpecName "kube-api-access-7ph9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:09:53.827263 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.827227 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-openshift-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:53.827263 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.827259 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ph9n\" (UniqueName: \"kubernetes.io/projected/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-kube-api-access-7ph9n\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:53.827454 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:53.827273 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7946c1ae-ab06-4d3c-b02a-a6534bb8a174-maas-api-xks-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:09:54.459181 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:54.459102 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" Nov 02 22:09:54.459181 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:54.459139 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj" event={"ID":"7946c1ae-ab06-4d3c-b02a-a6534bb8a174","Type":"ContainerDied","Data":"6f80d41981a8e1cf0bb55136cdb34d5af77287027f1c123fb41ebbe1a8ea0236"} Nov 02 22:09:54.459363 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:54.459186 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f80d41981a8e1cf0bb55136cdb34d5af77287027f1c123fb41ebbe1a8ea0236" Nov 02 22:09:54.700562 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:09:54.700508 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85"] Nov 02 22:10:00.823597 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.823563 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl"] Nov 02 22:10:00.824042 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.824029 4510 cpu_manager.go:414] "containerMap: removing container" podUID="7946c1ae-ab06-4d3c-b02a-a6534bb8a174" containerName="revoke-keys" Nov 02 22:10:00.824108 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.824043 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7946c1ae-ab06-4d3c-b02a-a6534bb8a174" containerName="revoke-keys" Nov 02 22:10:00.824146 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.824113 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="7946c1ae-ab06-4d3c-b02a-a6534bb8a174" containerName="revoke-keys" Nov 02 22:10:00.827254 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.827229 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:00.876224 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.876180 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:00.898866 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.898840 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:00.898984 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.898872 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgrz\" (UniqueName: \"kubernetes.io/projected/b82774dc-534e-4415-9cf0-4d73fe59bfaa-kube-api-access-ncgrz\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:00.898984 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.898900 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:00.922189 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.922163 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:10:00.924294 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:00.924270 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl"] Nov 02 22:10:01.000499 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.000461 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:01.000664 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.000549 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgrz\" (UniqueName: \"kubernetes.io/projected/b82774dc-534e-4415-9cf0-4d73fe59bfaa-kube-api-access-ncgrz\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:01.000664 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.000595 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:01.001061 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.001039 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:01.002860 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.002834 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:01.053150 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.053119 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgrz\" (UniqueName: \"kubernetes.io/projected/b82774dc-534e-4415-9cf0-4d73fe59bfaa-kube-api-access-ncgrz\") pod \"maas-api-revoke-sub-60942862-n6bxl\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:01.142698 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.142673 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:01.511202 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:01.511119 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl"] Nov 02 22:10:01.518836 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:10:01.518808 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82774dc_534e_4415_9cf0_4d73fe59bfaa.slice/crio-abff50a5f3228f088b0f4dafde8d2dc6fb13c2e39a0a40874ea8fe546ff106f7 WatchSource:0}: Error finding container abff50a5f3228f088b0f4dafde8d2dc6fb13c2e39a0a40874ea8fe546ff106f7: Status 404 returned error can't find the container with id abff50a5f3228f088b0f4dafde8d2dc6fb13c2e39a0a40874ea8fe546ff106f7 Nov 02 22:10:02.498809 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:02.498781 4510 generic.go:356] "Generic (PLEG): container finished" podID="b82774dc-534e-4415-9cf0-4d73fe59bfaa" containerID="554f979af37d2217f1693beaf90e73ab5054fb2ee56bcad1472e67e93fc9a7a1" exitCode=0 Nov 02 22:10:02.499127 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:02.498825 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" event={"ID":"b82774dc-534e-4415-9cf0-4d73fe59bfaa","Type":"ContainerDied","Data":"554f979af37d2217f1693beaf90e73ab5054fb2ee56bcad1472e67e93fc9a7a1"} Nov 02 22:10:02.499127 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:02.498871 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" event={"ID":"b82774dc-534e-4415-9cf0-4d73fe59bfaa","Type":"ContainerStarted","Data":"abff50a5f3228f088b0f4dafde8d2dc6fb13c2e39a0a40874ea8fe546ff106f7"} Nov 02 22:10:03.638192 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.638166 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:03.724560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.724510 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82774dc-534e-4415-9cf0-4d73fe59bfaa-kube-api-access-ncgrz\" (UniqueName: \"kubernetes.io/projected/b82774dc-534e-4415-9cf0-4d73fe59bfaa-kube-api-access-ncgrz\") pod \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " Nov 02 22:10:03.724764 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.724624 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-xks-ca\") pod \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " Nov 02 22:10:03.724764 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.724669 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-openshift-ca\") pod \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\" (UID: \"b82774dc-534e-4415-9cf0-4d73fe59bfaa\") " Nov 02 22:10:03.725064 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.725036 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-openshift-ca" pod "b82774dc-534e-4415-9cf0-4d73fe59bfaa" (UID: "b82774dc-534e-4415-9cf0-4d73fe59bfaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:03.726647 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.726628 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-xks-ca" pod "b82774dc-534e-4415-9cf0-4d73fe59bfaa" (UID: "b82774dc-534e-4415-9cf0-4d73fe59bfaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:03.726759 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.726740 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82774dc-534e-4415-9cf0-4d73fe59bfaa-kube-api-access-ncgrz" pod "b82774dc-534e-4415-9cf0-4d73fe59bfaa" (UID: "b82774dc-534e-4415-9cf0-4d73fe59bfaa"). InnerVolumeSpecName "kube-api-access-ncgrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:03.825807 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.825724 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncgrz\" (UniqueName: \"kubernetes.io/projected/b82774dc-534e-4415-9cf0-4d73fe59bfaa-kube-api-access-ncgrz\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:03.825807 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.825751 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-xks-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:03.825807 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:03.825762 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b82774dc-534e-4415-9cf0-4d73fe59bfaa-maas-api-openshift-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:04.509682 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:04.509595 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" event={"ID":"b82774dc-534e-4415-9cf0-4d73fe59bfaa","Type":"ContainerDied","Data":"abff50a5f3228f088b0f4dafde8d2dc6fb13c2e39a0a40874ea8fe546ff106f7"} Nov 02 22:10:04.509682 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:04.509630 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl" Nov 02 22:10:04.509682 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:04.509649 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abff50a5f3228f088b0f4dafde8d2dc6fb13c2e39a0a40874ea8fe546ff106f7" Nov 02 22:10:09.721302 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:09.721255 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" containerID="cri-o://17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0" gracePeriod=45 Nov 02 22:10:09.970972 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:09.970942 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 22:10:10.088573 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.088488 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e53dc8e1-072d-4586-a65a-9417b18bbe44-tls-certs\" (UniqueName: \"kubernetes.io/secret/e53dc8e1-072d-4586-a65a-9417b18bbe44-tls-certs\") pod \"e53dc8e1-072d-4586-a65a-9417b18bbe44\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " Nov 02 22:10:10.088573 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.088568 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-model-cache\") pod \"e53dc8e1-072d-4586-a65a-9417b18bbe44\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " Nov 02 22:10:10.088793 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.088615 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-tmp-dir\") pod \"e53dc8e1-072d-4586-a65a-9417b18bbe44\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " Nov 02 22:10:10.088793 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.088684 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e53dc8e1-072d-4586-a65a-9417b18bbe44-kube-api-access-js9mj\" (UniqueName: \"kubernetes.io/projected/e53dc8e1-072d-4586-a65a-9417b18bbe44-kube-api-access-js9mj\") pod \"e53dc8e1-072d-4586-a65a-9417b18bbe44\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " Nov 02 22:10:10.088793 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.088771 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-home\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-home\") pod \"e53dc8e1-072d-4586-a65a-9417b18bbe44\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " Nov 02 22:10:10.089000 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.088823 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-dshm\") pod \"e53dc8e1-072d-4586-a65a-9417b18bbe44\" (UID: \"e53dc8e1-072d-4586-a65a-9417b18bbe44\") " Nov 02 22:10:10.089000 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.088901 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-model-cache" pod "e53dc8e1-072d-4586-a65a-9417b18bbe44" (UID: "e53dc8e1-072d-4586-a65a-9417b18bbe44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.089160 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.089135 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-home" pod "e53dc8e1-072d-4586-a65a-9417b18bbe44" (UID: "e53dc8e1-072d-4586-a65a-9417b18bbe44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.089223 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.089186 4510 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-model-cache\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.089223 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.089203 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-tmp-dir" pod "e53dc8e1-072d-4586-a65a-9417b18bbe44" (UID: "e53dc8e1-072d-4586-a65a-9417b18bbe44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.091070 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.091041 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-dshm" pod "e53dc8e1-072d-4586-a65a-9417b18bbe44" (UID: "e53dc8e1-072d-4586-a65a-9417b18bbe44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:10:10.091169 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.091151 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53dc8e1-072d-4586-a65a-9417b18bbe44-tls-certs" pod "e53dc8e1-072d-4586-a65a-9417b18bbe44" (UID: "e53dc8e1-072d-4586-a65a-9417b18bbe44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:10.095631 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.095608 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53dc8e1-072d-4586-a65a-9417b18bbe44-kube-api-access-js9mj" pod "e53dc8e1-072d-4586-a65a-9417b18bbe44" (UID: "e53dc8e1-072d-4586-a65a-9417b18bbe44"). InnerVolumeSpecName "kube-api-access-js9mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:10.190371 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.190339 4510 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e53dc8e1-072d-4586-a65a-9417b18bbe44-tls-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.190371 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.190368 4510 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-tmp-dir\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.190371 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.190379 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9mj\" (UniqueName: \"kubernetes.io/projected/e53dc8e1-072d-4586-a65a-9417b18bbe44-kube-api-access-js9mj\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.190619 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.190389 4510 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-home\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.190619 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.190397 4510 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e53dc8e1-072d-4586-a65a-9417b18bbe44-dshm\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:10.539185 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.539151 4510 generic.go:356] "Generic (PLEG): container finished" podID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerID="17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0" exitCode=0 Nov 02 22:10:10.539372 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.539198 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" Nov 02 22:10:10.539372 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.539208 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" event={"ID":"e53dc8e1-072d-4586-a65a-9417b18bbe44","Type":"ContainerDied","Data":"17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0"} Nov 02 22:10:10.539372 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.539247 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85" event={"ID":"e53dc8e1-072d-4586-a65a-9417b18bbe44","Type":"ContainerDied","Data":"1e9099cb6a4b64158d8dcf2a3ec4847e7befe04db0025f2a30964f07726d3403"} Nov 02 22:10:10.539372 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.539267 4510 scope.go:122] "RemoveContainer" containerID="17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0" Nov 02 22:10:10.548122 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.548105 4510 scope.go:122] "RemoveContainer" containerID="17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0" Nov 02 22:10:10.548386 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:10:10.548366 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0\": container with ID starting with 17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0 not found: ID does not exist" containerID="17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0" Nov 02 22:10:10.548424 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.548395 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0"} err="failed to get container status \"17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0\": rpc error: code = NotFound desc = could not find container \"17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0\": container with ID starting with 17b06ccda820e75d5514ffe232353b03cb7270669547e77a88ce1b3fa89e44d0 not found: ID does not exist" Nov 02 22:10:10.689847 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.689813 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85"] Nov 02 22:10:10.735594 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:10.735560 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-284b8e/distinct-w6-284b8e-kserve-7dc855589b-qpd85"] Nov 02 22:10:12.152561 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:12.152514 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" path="/var/lib/kubelet/pods/e53dc8e1-072d-4586-a65a-9417b18bbe44/volumes" Nov 02 22:10:15.533213 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:10:15.533176 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:10:16.877258 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:16.877225 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj"] Nov 02 22:10:16.923247 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:16.923216 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-284b8e-jvlwj"] Nov 02 22:10:17.288596 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:17.288507 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:10:18.153064 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:18.153032 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7946c1ae-ab06-4d3c-b02a-a6534bb8a174" path="/var/lib/kubelet/pods/7946c1ae-ab06-4d3c-b02a-a6534bb8a174/volumes" Nov 02 22:10:30.918163 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:30.918129 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:10:45.531380 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:10:45.531288 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:10:59.538747 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:10:59.538710 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:11:15.529226 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:11:15.529190 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:11:45.530005 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:11:45.529969 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:11:52.320859 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:11:52.320823 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:12:00.739648 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.739611 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2"] Nov 02 22:12:00.740110 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.740096 4510 cpu_manager.go:414] "containerMap: removing container" podUID="b82774dc-534e-4415-9cf0-4d73fe59bfaa" containerName="revoke-keys" Nov 02 22:12:00.740161 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.740112 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82774dc-534e-4415-9cf0-4d73fe59bfaa" containerName="revoke-keys" Nov 02 22:12:00.740161 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.740146 4510 cpu_manager.go:414] "containerMap: removing container" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" Nov 02 22:12:00.740161 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.740154 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" Nov 02 22:12:00.740331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.740233 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="e53dc8e1-072d-4586-a65a-9417b18bbe44" containerName="main" Nov 02 22:12:00.740331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.740244 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82774dc-534e-4415-9cf0-4d73fe59bfaa" containerName="revoke-keys" Nov 02 22:12:00.743457 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.743426 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.794597 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.794563 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:00.833073 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.833038 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.833073 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.833077 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4kg\" (UniqueName: \"kubernetes.io/projected/19821182-4a56-4e53-b19a-1e401726e83c-kube-api-access-8f4kg\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.833282 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.833174 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.840769 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.840740 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:12:00.842671 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.842653 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2"] Nov 02 22:12:00.934348 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.934322 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.934477 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.934353 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4kg\" (UniqueName: \"kubernetes.io/projected/19821182-4a56-4e53-b19a-1e401726e83c-kube-api-access-8f4kg\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.934581 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.934563 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.935174 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.935153 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.936600 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.936584 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:00.986827 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:00.986802 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4kg\" (UniqueName: \"kubernetes.io/projected/19821182-4a56-4e53-b19a-1e401726e83c-kube-api-access-8f4kg\") pod \"maas-api-revoke-sub-c1e271a9-hf2k2\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:01.057543 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:01.057466 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:01.426057 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:01.426025 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2"] Nov 02 22:12:01.433754 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:12:01.433729 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19821182_4a56_4e53_b19a_1e401726e83c.slice/crio-f4c91d7b5926420387e35af61f3ebf51050f9c859ed5ab4401e21bca52ced6b9 WatchSource:0}: Error finding container f4c91d7b5926420387e35af61f3ebf51050f9c859ed5ab4401e21bca52ced6b9: Status 404 returned error can't find the container with id f4c91d7b5926420387e35af61f3ebf51050f9c859ed5ab4401e21bca52ced6b9 Nov 02 22:12:02.079437 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:02.079344 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" event={"ID":"19821182-4a56-4e53-b19a-1e401726e83c","Type":"ContainerStarted","Data":"27abf4f064d6fbe079dfdefa4277b36f9f90bae75ede3c5b2108aea3b5e966ab"} Nov 02 22:12:02.079437 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:02.079411 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" event={"ID":"19821182-4a56-4e53-b19a-1e401726e83c","Type":"ContainerStarted","Data":"f4c91d7b5926420387e35af61f3ebf51050f9c859ed5ab4401e21bca52ced6b9"} Nov 02 22:12:02.186608 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:02.186544 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" podStartSLOduration=2.186531597 podStartE2EDuration="2.186531597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:02.184545751 +0000 UTC m=+3548.160843522" watchObservedRunningTime="2026-11-02 22:12:02.186531597 +0000 UTC m=+3548.162829364" Nov 02 22:12:03.085502 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:03.085461 4510 generic.go:356] "Generic (PLEG): container finished" podID="19821182-4a56-4e53-b19a-1e401726e83c" containerID="27abf4f064d6fbe079dfdefa4277b36f9f90bae75ede3c5b2108aea3b5e966ab" exitCode=0 Nov 02 22:12:03.085914 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:03.085548 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" event={"ID":"19821182-4a56-4e53-b19a-1e401726e83c","Type":"ContainerDied","Data":"27abf4f064d6fbe079dfdefa4277b36f9f90bae75ede3c5b2108aea3b5e966ab"} Nov 02 22:12:04.224110 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.224085 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:04.366684 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.366656 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-openshift-ca\") pod \"19821182-4a56-4e53-b19a-1e401726e83c\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " Nov 02 22:12:04.366872 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.366743 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19821182-4a56-4e53-b19a-1e401726e83c-kube-api-access-8f4kg\" (UniqueName: \"kubernetes.io/projected/19821182-4a56-4e53-b19a-1e401726e83c-kube-api-access-8f4kg\") pod \"19821182-4a56-4e53-b19a-1e401726e83c\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " Nov 02 22:12:04.366872 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.366815 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-xks-ca\") pod \"19821182-4a56-4e53-b19a-1e401726e83c\" (UID: \"19821182-4a56-4e53-b19a-1e401726e83c\") " Nov 02 22:12:04.367094 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.367069 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-openshift-ca" pod "19821182-4a56-4e53-b19a-1e401726e83c" (UID: "19821182-4a56-4e53-b19a-1e401726e83c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:04.368853 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.368827 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19821182-4a56-4e53-b19a-1e401726e83c-kube-api-access-8f4kg" pod "19821182-4a56-4e53-b19a-1e401726e83c" (UID: "19821182-4a56-4e53-b19a-1e401726e83c"). InnerVolumeSpecName "kube-api-access-8f4kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:04.368947 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.368855 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-xks-ca" pod "19821182-4a56-4e53-b19a-1e401726e83c" (UID: "19821182-4a56-4e53-b19a-1e401726e83c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:04.467757 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.467723 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-openshift-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.467757 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.467754 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f4kg\" (UniqueName: \"kubernetes.io/projected/19821182-4a56-4e53-b19a-1e401726e83c-kube-api-access-8f4kg\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:04.467757 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:04.467764 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19821182-4a56-4e53-b19a-1e401726e83c-maas-api-xks-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:05.096052 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:05.096027 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" Nov 02 22:12:05.096225 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:05.096021 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2" event={"ID":"19821182-4a56-4e53-b19a-1e401726e83c","Type":"ContainerDied","Data":"f4c91d7b5926420387e35af61f3ebf51050f9c859ed5ab4401e21bca52ced6b9"} Nov 02 22:12:05.096225 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:05.096155 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c91d7b5926420387e35af61f3ebf51050f9c859ed5ab4401e21bca52ced6b9" Nov 02 22:12:12.044156 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.044122 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb"] Nov 02 22:12:12.044606 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.044590 4510 cpu_manager.go:414] "containerMap: removing container" podUID="19821182-4a56-4e53-b19a-1e401726e83c" containerName="revoke-keys" Nov 02 22:12:12.044660 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.044606 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19821182-4a56-4e53-b19a-1e401726e83c" containerName="revoke-keys" Nov 02 22:12:12.044709 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.044698 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="19821182-4a56-4e53-b19a-1e401726e83c" containerName="revoke-keys" Nov 02 22:12:12.048120 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.048101 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.097037 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.097013 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:12:12.097146 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.097081 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-445mt\"" Nov 02 22:12:12.140832 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.140805 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.140952 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.140850 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cbd378-8dec-4cb9-9a52-f145dec09563-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.141026 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.140954 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.141026 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.141011 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.141121 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.141102 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.141157 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.141145 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8qc\" (UniqueName: \"kubernetes.io/projected/63cbd378-8dec-4cb9-9a52-f145dec09563-kube-api-access-6q8qc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.141247 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.141230 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.141289 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.141256 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.141289 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.141279 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.145880 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.145856 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb"] Nov 02 22:12:12.242571 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.242546 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cbd378-8dec-4cb9-9a52-f145dec09563-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.242739 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.242597 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.242739 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.242645 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.242739 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.242688 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.242739 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.242735 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8qc\" (UniqueName: \"kubernetes.io/projected/63cbd378-8dec-4cb9-9a52-f145dec09563-kube-api-access-6q8qc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243014 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.242992 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243102 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243074 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243102 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243080 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243366 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243337 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243366 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243358 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243478 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243389 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243478 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243359 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cbd378-8dec-4cb9-9a52-f145dec09563-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243642 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243623 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.243718 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.243697 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.245211 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.245187 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.245291 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.245238 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.297806 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.297743 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cbd378-8dec-4cb9-9a52-f145dec09563-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.297902 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.297849 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8qc\" (UniqueName: \"kubernetes.io/projected/63cbd378-8dec-4cb9-9a52-f145dec09563-kube-api-access-6q8qc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-kqfwb\" (UID: \"63cbd378-8dec-4cb9-9a52-f145dec09563\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.361182 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.361153 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:12.785361 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.785331 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb"] Nov 02 22:12:12.794185 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:12:12.794158 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cbd378_8dec_4cb9_9a52_f145dec09563.slice/crio-aec5f01070e0f6ed5281c649b14486bfb57109f9215047f58eead6bbd38868d9 WatchSource:0}: Error finding container aec5f01070e0f6ed5281c649b14486bfb57109f9215047f58eead6bbd38868d9: Status 404 returned error can't find the container with id aec5f01070e0f6ed5281c649b14486bfb57109f9215047f58eead6bbd38868d9 Nov 02 22:12:12.796397 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.796365 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:12.796471 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.796436 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:12.796471 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:12.796468 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:13.137990 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:13.137954 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" event={"ID":"63cbd378-8dec-4cb9-9a52-f145dec09563","Type":"ContainerStarted","Data":"ee3dfc2b6dbb961cbdd4dbe1f06576e58d619bf013256015cb2c97b591cad2eb"} Nov 02 22:12:13.138405 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:13.138027 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" event={"ID":"63cbd378-8dec-4cb9-9a52-f145dec09563","Type":"ContainerStarted","Data":"aec5f01070e0f6ed5281c649b14486bfb57109f9215047f58eead6bbd38868d9"} Nov 02 22:12:13.247010 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:13.246958 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podStartSLOduration=1.246946334 podStartE2EDuration="1.246946334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:13.245816236 +0000 UTC m=+3559.222114015" watchObservedRunningTime="2026-11-02 22:12:13.246946334 +0000 UTC m=+3559.223244108" Nov 02 22:12:13.361600 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:13.361558 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:13.363205 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:13.363176 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 22:12:13.363425 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:13.363244 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 22:12:13.706165 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:13.706133 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:12:14.362055 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:14.362019 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 22:12:14.362474 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:14.362069 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 22:12:15.362036 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:15.361998 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 22:12:15.362217 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:15.362053 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 22:12:15.554632 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:12:15.554595 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:12:16.361787 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:16.361749 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 22:12:16.361995 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:16.361810 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 22:12:17.362230 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:17.362189 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 22:12:17.362760 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:17.362247 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 22:12:18.362419 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:18.362389 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.362872 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:18.362439 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 22:12:19.361695 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:19.361659 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 22:12:19.361873 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:19.361723 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 22:12:21.362474 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:21.362430 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:21.362932 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:21.362498 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.403684 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:22.403644 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:22.404100 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:22.403693 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.404667 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:23.404619 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:23.405170 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:23.404691 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.821573 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:23.821452 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:12:24.446728 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:24.446679 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:24.447279 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:24.446761 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:25.053471 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:25.053440 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:25.053786 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:25.053767 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:25.054580 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:25.054560 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" Nov 02 22:12:27.163757 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.163726 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx"] Nov 02 22:12:27.167660 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.167634 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.259301 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.259269 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx"] Nov 02 22:12:27.291022 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.290986 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291182 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291035 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291182 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291066 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e012812-5cc5-451d-94d6-75adbdc21fff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291182 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291150 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6c8\" (UniqueName: \"kubernetes.io/projected/4e012812-5cc5-451d-94d6-75adbdc21fff-kube-api-access-ll6c8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291304 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291229 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291304 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291289 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291382 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291359 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291418 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291397 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.291462 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.291430 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392544 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392490 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392696 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392604 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392696 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392640 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392696 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392671 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392839 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392713 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392839 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392759 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392839 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392800 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.392839 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392827 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e012812-5cc5-451d-94d6-75adbdc21fff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.393058 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392857 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6c8\" (UniqueName: \"kubernetes.io/projected/4e012812-5cc5-451d-94d6-75adbdc21fff-kube-api-access-ll6c8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.393058 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.392967 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.393214 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.393190 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.393279 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.393229 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.393337 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.393314 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.393541 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.393504 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e012812-5cc5-451d-94d6-75adbdc21fff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.395073 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.395055 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.395465 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.395445 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.445760 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.445693 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e012812-5cc5-451d-94d6-75adbdc21fff-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.445926 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.445907 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6c8\" (UniqueName: \"kubernetes.io/projected/4e012812-5cc5-451d-94d6-75adbdc21fff-kube-api-access-ll6c8\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-zndhx\" (UID: \"4e012812-5cc5-451d-94d6-75adbdc21fff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.480469 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.480442 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:27.902796 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.902767 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx"] Nov 02 22:12:27.910173 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:12:27.910149 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e012812_5cc5_451d_94d6_75adbdc21fff.slice/crio-8158737be521995cc3c33c01bc44e4f65ec1369ae1642a68afb26acf0d911982 WatchSource:0}: Error finding container 8158737be521995cc3c33c01bc44e4f65ec1369ae1642a68afb26acf0d911982: Status 404 returned error can't find the container with id 8158737be521995cc3c33c01bc44e4f65ec1369ae1642a68afb26acf0d911982 Nov 02 22:12:27.912377 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.912350 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:27.912438 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.912412 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:27.912474 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:27.912441 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:12:28.225185 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:28.225108 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" event={"ID":"4e012812-5cc5-451d-94d6-75adbdc21fff","Type":"ContainerStarted","Data":"44c2aa1b42c188bad89a1c93e735584f8762435fbd51219a282b7a6a976c043e"} Nov 02 22:12:28.225185 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:28.225175 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" event={"ID":"4e012812-5cc5-451d-94d6-75adbdc21fff","Type":"ContainerStarted","Data":"8158737be521995cc3c33c01bc44e4f65ec1369ae1642a68afb26acf0d911982"} Nov 02 22:12:28.481642 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:28.481553 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:28.483264 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:28.483242 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:12:28.483380 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:28.483287 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 22:12:29.480877 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:29.480841 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:12:29.481270 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:29.480897 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 22:12:30.481618 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:30.481582 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:12:30.482008 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:30.481633 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 22:12:31.481732 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:31.481696 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:12:31.482147 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:31.481755 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 22:12:32.481162 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:32.481127 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:12:32.481346 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:32.481192 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 22:12:33.481149 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:33.481112 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:12:33.481553 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:33.481166 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 22:12:34.481922 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:34.481881 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 22:12:34.482308 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:34.481955 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 22:12:35.965239 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:35.965193 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:35.965734 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:35.965510 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:35.966641 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:35.966618 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" Nov 02 22:12:36.121045 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:36.120990 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podStartSLOduration=9.120977404 podStartE2EDuration="9.120977404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:28.336645488 +0000 UTC m=+3574.312943266" watchObservedRunningTime="2026-11-02 22:12:36.120977404 +0000 UTC m=+3582.097275157" Nov 02 22:12:45.532388 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:12:45.532352 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:12:48.580032 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:12:48.579999 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:13:01.507793 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:01.507754 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:13:15.529735 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:13:15.529691 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:13:18.987227 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:18.987194 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb"] Nov 02 22:13:18.990666 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:18.990647 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.039925 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.039901 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2pd\" (UniqueName: \"kubernetes.io/projected/f609a3b6-ee87-42cb-9d9b-1a29296b4103-kube-api-access-nt2pd\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.040044 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.039936 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.040044 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.039954 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.041780 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.041760 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:19.042052 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.042034 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:13:19.044224 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.044202 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb"] Nov 02 22:13:19.140573 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.140514 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2pd\" (UniqueName: \"kubernetes.io/projected/f609a3b6-ee87-42cb-9d9b-1a29296b4103-kube-api-access-nt2pd\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.140736 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.140599 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.140736 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.140627 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.141313 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.141293 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.142899 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.142875 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.194993 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.194970 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2pd\" (UniqueName: \"kubernetes.io/projected/f609a3b6-ee87-42cb-9d9b-1a29296b4103-kube-api-access-nt2pd\") pod \"maas-api-revoke-sub-046e1343-l96qb\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.303844 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.303783 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:19.671971 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:19.671941 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb"] Nov 02 22:13:19.679555 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:13:19.679514 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf609a3b6_ee87_42cb_9d9b_1a29296b4103.slice/crio-5e6d21232a3d324604ae33365e73bef9ca92ef47444657744d56efe7abee572b WatchSource:0}: Error finding container 5e6d21232a3d324604ae33365e73bef9ca92ef47444657744d56efe7abee572b: Status 404 returned error can't find the container with id 5e6d21232a3d324604ae33365e73bef9ca92ef47444657744d56efe7abee572b Nov 02 22:13:20.507572 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:20.507513 4510 generic.go:356] "Generic (PLEG): container finished" podID="f609a3b6-ee87-42cb-9d9b-1a29296b4103" containerID="791419040df6dfa95d94b98aa6dab9a499ab985fe78291eb21f6486642c5e492" exitCode=0 Nov 02 22:13:20.507889 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:20.507621 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" event={"ID":"f609a3b6-ee87-42cb-9d9b-1a29296b4103","Type":"ContainerDied","Data":"791419040df6dfa95d94b98aa6dab9a499ab985fe78291eb21f6486642c5e492"} Nov 02 22:13:20.507889 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:20.507677 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" event={"ID":"f609a3b6-ee87-42cb-9d9b-1a29296b4103","Type":"ContainerStarted","Data":"5e6d21232a3d324604ae33365e73bef9ca92ef47444657744d56efe7abee572b"} Nov 02 22:13:21.644975 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.644950 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:21.666124 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.666098 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-xks-ca\") pod \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " Nov 02 22:13:21.666237 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.666138 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-openshift-ca\") pod \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " Nov 02 22:13:21.666312 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.666300 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f609a3b6-ee87-42cb-9d9b-1a29296b4103-kube-api-access-nt2pd\" (UniqueName: \"kubernetes.io/projected/f609a3b6-ee87-42cb-9d9b-1a29296b4103-kube-api-access-nt2pd\") pod \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\" (UID: \"f609a3b6-ee87-42cb-9d9b-1a29296b4103\") " Nov 02 22:13:21.666488 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.666466 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-openshift-ca" pod "f609a3b6-ee87-42cb-9d9b-1a29296b4103" (UID: "f609a3b6-ee87-42cb-9d9b-1a29296b4103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:21.668147 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.668120 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-xks-ca" pod "f609a3b6-ee87-42cb-9d9b-1a29296b4103" (UID: "f609a3b6-ee87-42cb-9d9b-1a29296b4103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:21.668242 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.668172 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f609a3b6-ee87-42cb-9d9b-1a29296b4103-kube-api-access-nt2pd" pod "f609a3b6-ee87-42cb-9d9b-1a29296b4103" (UID: "f609a3b6-ee87-42cb-9d9b-1a29296b4103"). InnerVolumeSpecName "kube-api-access-nt2pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:21.767927 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.767854 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-openshift-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.767927 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.767882 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt2pd\" (UniqueName: \"kubernetes.io/projected/f609a3b6-ee87-42cb-9d9b-1a29296b4103-kube-api-access-nt2pd\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:21.767927 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:21.767892 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f609a3b6-ee87-42cb-9d9b-1a29296b4103-maas-api-xks-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:22.518310 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:22.518234 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" Nov 02 22:13:22.518310 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:22.518260 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb" event={"ID":"f609a3b6-ee87-42cb-9d9b-1a29296b4103","Type":"ContainerDied","Data":"5e6d21232a3d324604ae33365e73bef9ca92ef47444657744d56efe7abee572b"} Nov 02 22:13:22.518310 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:22.518302 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e6d21232a3d324604ae33365e73bef9ca92ef47444657744d56efe7abee572b" Nov 02 22:13:27.582367 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.582333 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn"] Nov 02 22:13:27.582814 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.582800 4510 cpu_manager.go:414] "containerMap: removing container" podUID="f609a3b6-ee87-42cb-9d9b-1a29296b4103" containerName="revoke-keys" Nov 02 22:13:27.582814 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.582814 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f609a3b6-ee87-42cb-9d9b-1a29296b4103" containerName="revoke-keys" Nov 02 22:13:27.582893 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.582884 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="f609a3b6-ee87-42cb-9d9b-1a29296b4103" containerName="revoke-keys" Nov 02 22:13:27.586176 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.586157 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623333 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623306 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623446 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623357 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623488 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623431 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623488 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623484 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623611 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623509 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623682 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623656 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623743 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623734 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623785 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623773 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfjp2\" (UniqueName: \"kubernetes.io/projected/4696f74b-e8cf-4518-96f7-c6b29e686be5-kube-api-access-zfjp2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.623831 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.623793 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4696f74b-e8cf-4518-96f7-c6b29e686be5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.683949 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.683920 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn"] Nov 02 22:13:27.725266 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725233 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725421 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725285 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725421 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725328 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725421 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725409 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725452 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725513 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725581 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfjp2\" (UniqueName: \"kubernetes.io/projected/4696f74b-e8cf-4518-96f7-c6b29e686be5-kube-api-access-zfjp2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725612 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725608 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4696f74b-e8cf-4518-96f7-c6b29e686be5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.725858 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725678 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.726013 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.725967 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.726098 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.726037 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.726098 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.726038 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.726220 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.726094 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.726618 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.726591 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4696f74b-e8cf-4518-96f7-c6b29e686be5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.727953 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.727926 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.728068 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.728051 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.778898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.778871 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4696f74b-e8cf-4518-96f7-c6b29e686be5-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.779034 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.778992 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfjp2\" (UniqueName: \"kubernetes.io/projected/4696f74b-e8cf-4518-96f7-c6b29e686be5-kube-api-access-zfjp2\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-ndztn\" (UID: \"4696f74b-e8cf-4518-96f7-c6b29e686be5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:27.899077 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:27.899045 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:28.323082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.323002 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn"] Nov 02 22:13:28.330794 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:13:28.330759 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4696f74b_e8cf_4518_96f7_c6b29e686be5.slice/crio-51290acb731703d4b4bfc6d8f1554ce757e9cdeda55854cc4ab3c850f27c7ab9 WatchSource:0}: Error finding container 51290acb731703d4b4bfc6d8f1554ce757e9cdeda55854cc4ab3c850f27c7ab9: Status 404 returned error can't find the container with id 51290acb731703d4b4bfc6d8f1554ce757e9cdeda55854cc4ab3c850f27c7ab9 Nov 02 22:13:28.332867 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.332834 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:28.332977 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.332937 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:28.333036 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.332984 4510 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 22:13:28.549431 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.549390 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" event={"ID":"4696f74b-e8cf-4518-96f7-c6b29e686be5","Type":"ContainerStarted","Data":"8e9400daeaa105ed63c18772128f9deafc461fa6224e2a42896dee0686a28767"} Nov 02 22:13:28.549615 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.549458 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" event={"ID":"4696f74b-e8cf-4518-96f7-c6b29e686be5","Type":"ContainerStarted","Data":"51290acb731703d4b4bfc6d8f1554ce757e9cdeda55854cc4ab3c850f27c7ab9"} Nov 02 22:13:28.660566 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.660496 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podStartSLOduration=1.660481466 podStartE2EDuration="1.660481466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:28.657701555 +0000 UTC m=+3634.633999330" watchObservedRunningTime="2026-11-02 22:13:28.660481466 +0000 UTC m=+3634.636779241" Nov 02 22:13:28.900159 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:28.900127 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:29.901095 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:29.901059 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:13:29.901597 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:29.901118 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 22:13:30.899835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:30.899803 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:13:30.900015 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:30.899855 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 22:13:31.900352 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:31.900318 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:13:31.900755 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:31.900377 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 22:13:32.227546 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:32.223995 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:13:32.900083 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:32.900039 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:13:32.900252 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:32.900106 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 22:13:33.899570 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:33.899538 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:13:33.900186 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:33.900138 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 22:13:34.899644 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:34.899611 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:13:34.900042 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:34.899668 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 22:13:35.899613 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:35.899507 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 22:13:35.899613 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:35.899595 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 22:13:37.941692 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:37.941651 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:37.942118 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:37.941732 4510 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.779376 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:38.779346 4510 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:38.779739 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:38.779709 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:38.780390 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:38.780371 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" Nov 02 22:13:43.217560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:43.217516 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:13:45.531054 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:13:45.531013 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:13:51.781508 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:13:51.781469 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:14:15.529950 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:14:15.529910 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:14:19.211512 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:14:19.211477 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:14:22.013688 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:14:22.013646 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-zndhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:22.014117 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:14:22.013697 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-zndhx" podUID="4e012812-5cc5-451d-94d6-75adbdc21fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:41.812632 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:14:41.812601 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:14:45.532337 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:14:45.532300 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:14:54.907664 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:14:54.907630 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:15:00.161203 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.161174 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477"] Nov 02 22:15:00.164961 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.164940 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.211332 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.211305 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4lz\" (UniqueName: \"kubernetes.io/projected/dc88dae8-c9b6-45ba-b8e3-a08feacca859-kube-api-access-wd4lz\") pod \"maas-api-key-cleanup-29894295-74477\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.211481 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.211345 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc88dae8-c9b6-45ba-b8e3-a08feacca859-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-74477\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.212892 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.212870 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:15:00.262853 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.262817 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477"] Nov 02 22:15:00.312958 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.312915 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4lz\" (UniqueName: \"kubernetes.io/projected/dc88dae8-c9b6-45ba-b8e3-a08feacca859-kube-api-access-wd4lz\") pod \"maas-api-key-cleanup-29894295-74477\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.312958 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.312958 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc88dae8-c9b6-45ba-b8e3-a08feacca859-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-74477\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.313570 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.313552 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc88dae8-c9b6-45ba-b8e3-a08feacca859-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-74477\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.365931 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.365899 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4lz\" (UniqueName: \"kubernetes.io/projected/dc88dae8-c9b6-45ba-b8e3-a08feacca859-kube-api-access-wd4lz\") pod \"maas-api-key-cleanup-29894295-74477\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.477904 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.477837 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:00.845583 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:00.845488 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477"] Nov 02 22:15:00.852779 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:15:00.852749 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc88dae8_c9b6_45ba_b8e3_a08feacca859.slice/crio-58841f1c8663c7794cfe85b5ebf0037c80369de157fe540225178252cb101859 WatchSource:0}: Error finding container 58841f1c8663c7794cfe85b5ebf0037c80369de157fe540225178252cb101859: Status 404 returned error can't find the container with id 58841f1c8663c7794cfe85b5ebf0037c80369de157fe540225178252cb101859 Nov 02 22:15:01.143308 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:01.143273 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" event={"ID":"dc88dae8-c9b6-45ba-b8e3-a08feacca859","Type":"ContainerStarted","Data":"7ddba0eddc9c23f09ebcf91979a72670f4edbe6c333d76418006bc2d6687bbfb"} Nov 02 22:15:01.143420 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:01.143340 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" event={"ID":"dc88dae8-c9b6-45ba-b8e3-a08feacca859","Type":"ContainerStarted","Data":"58841f1c8663c7794cfe85b5ebf0037c80369de157fe540225178252cb101859"} Nov 02 22:15:01.247330 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:01.247267 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" podStartSLOduration=1.247241489 podStartE2EDuration="1.247241489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:01.246456481 +0000 UTC m=+3727.222754256" watchObservedRunningTime="2026-11-02 22:15:01.247241489 +0000 UTC m=+3727.223539269" Nov 02 22:15:02.150047 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:02.150009 4510 generic.go:356] "Generic (PLEG): container finished" podID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerID="7ddba0eddc9c23f09ebcf91979a72670f4edbe6c333d76418006bc2d6687bbfb" exitCode=60 Nov 02 22:15:02.150368 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:02.150351 4510 scope.go:122] "RemoveContainer" containerID="7ddba0eddc9c23f09ebcf91979a72670f4edbe6c333d76418006bc2d6687bbfb" Nov 02 22:15:02.152199 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:02.152179 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" event={"ID":"dc88dae8-c9b6-45ba-b8e3-a08feacca859","Type":"ContainerDied","Data":"7ddba0eddc9c23f09ebcf91979a72670f4edbe6c333d76418006bc2d6687bbfb"} Nov 02 22:15:03.156344 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:03.156307 4510 generic.go:356] "Generic (PLEG): container finished" podID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerID="504b61231e07b599055dde0d83368139dac3deb3abdd4fb846e8b87a28e1e9ae" exitCode=60 Nov 02 22:15:03.156750 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:03.156389 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" event={"ID":"dc88dae8-c9b6-45ba-b8e3-a08feacca859","Type":"ContainerDied","Data":"504b61231e07b599055dde0d83368139dac3deb3abdd4fb846e8b87a28e1e9ae"} Nov 02 22:15:03.156750 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:03.156451 4510 scope.go:122] "RemoveContainer" containerID="7ddba0eddc9c23f09ebcf91979a72670f4edbe6c333d76418006bc2d6687bbfb" Nov 02 22:15:03.156750 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:03.156682 4510 scope.go:122] "RemoveContainer" containerID="504b61231e07b599055dde0d83368139dac3deb3abdd4fb846e8b87a28e1e9ae" Nov 02 22:15:03.156970 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:15:03.156952 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-74477_odh-ai-gateway-infra(dc88dae8-c9b6-45ba-b8e3-a08feacca859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" Nov 02 22:15:04.045375 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:04.045340 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl"] Nov 02 22:15:04.091166 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:04.091133 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60942862-n6bxl"] Nov 02 22:15:04.160313 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:04.160279 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82774dc-534e-4415-9cf0-4d73fe59bfaa" path="/var/lib/kubelet/pods/b82774dc-534e-4415-9cf0-4d73fe59bfaa/volumes" Nov 02 22:15:04.162670 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:04.162652 4510 scope.go:122] "RemoveContainer" containerID="504b61231e07b599055dde0d83368139dac3deb3abdd4fb846e8b87a28e1e9ae" Nov 02 22:15:04.162837 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:15:04.162820 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-74477_odh-ai-gateway-infra(dc88dae8-c9b6-45ba-b8e3-a08feacca859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" Nov 02 22:15:14.924675 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:14.924593 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:15:15.534373 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:15:15.534330 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:15:16.146840 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:16.146796 4510 scope.go:122] "RemoveContainer" containerID="504b61231e07b599055dde0d83368139dac3deb3abdd4fb846e8b87a28e1e9ae" Nov 02 22:15:17.276627 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:17.276586 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" event={"ID":"dc88dae8-c9b6-45ba-b8e3-a08feacca859","Type":"ContainerStarted","Data":"a3fa36ed4d97e05db47a9c9889f176d414bac8f73c7a26544bd15738e937b65f"} Nov 02 22:15:18.283086 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:18.283055 4510 generic.go:356] "Generic (PLEG): container finished" podID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerID="a3fa36ed4d97e05db47a9c9889f176d414bac8f73c7a26544bd15738e937b65f" exitCode=60 Nov 02 22:15:18.283512 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:18.283128 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" event={"ID":"dc88dae8-c9b6-45ba-b8e3-a08feacca859","Type":"ContainerDied","Data":"a3fa36ed4d97e05db47a9c9889f176d414bac8f73c7a26544bd15738e937b65f"} Nov 02 22:15:18.283512 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:18.283186 4510 scope.go:122] "RemoveContainer" containerID="504b61231e07b599055dde0d83368139dac3deb3abdd4fb846e8b87a28e1e9ae" Nov 02 22:15:18.283676 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:18.283562 4510 scope.go:122] "RemoveContainer" containerID="a3fa36ed4d97e05db47a9c9889f176d414bac8f73c7a26544bd15738e937b65f" Nov 02 22:15:18.283835 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:15:18.283817 4510 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-74477_odh-ai-gateway-infra(dc88dae8-c9b6-45ba-b8e3-a08feacca859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" Nov 02 22:15:18.411298 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:18.411265 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477"] Nov 02 22:15:19.430050 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:19.430028 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:19.530484 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:19.530456 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc88dae8-c9b6-45ba-b8e3-a08feacca859-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc88dae8-c9b6-45ba-b8e3-a08feacca859-maas-api-service-ca\") pod \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " Nov 02 22:15:19.530653 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:19.530571 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc88dae8-c9b6-45ba-b8e3-a08feacca859-kube-api-access-wd4lz\" (UniqueName: \"kubernetes.io/projected/dc88dae8-c9b6-45ba-b8e3-a08feacca859-kube-api-access-wd4lz\") pod \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\" (UID: \"dc88dae8-c9b6-45ba-b8e3-a08feacca859\") " Nov 02 22:15:19.530796 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:19.530773 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc88dae8-c9b6-45ba-b8e3-a08feacca859-maas-api-service-ca" pod "dc88dae8-c9b6-45ba-b8e3-a08feacca859" (UID: "dc88dae8-c9b6-45ba-b8e3-a08feacca859"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:19.532621 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:19.532601 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc88dae8-c9b6-45ba-b8e3-a08feacca859-kube-api-access-wd4lz" pod "dc88dae8-c9b6-45ba-b8e3-a08feacca859" (UID: "dc88dae8-c9b6-45ba-b8e3-a08feacca859"). InnerVolumeSpecName "kube-api-access-wd4lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:19.631854 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:19.631822 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc88dae8-c9b6-45ba-b8e3-a08feacca859-maas-api-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.631854 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:19.631849 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4lz\" (UniqueName: \"kubernetes.io/projected/dc88dae8-c9b6-45ba-b8e3-a08feacca859-kube-api-access-wd4lz\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.295733 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:20.295707 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" Nov 02 22:15:20.295925 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:20.295733 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477" event={"ID":"dc88dae8-c9b6-45ba-b8e3-a08feacca859","Type":"ContainerDied","Data":"58841f1c8663c7794cfe85b5ebf0037c80369de157fe540225178252cb101859"} Nov 02 22:15:20.295925 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:20.295784 4510 scope.go:122] "RemoveContainer" containerID="a3fa36ed4d97e05db47a9c9889f176d414bac8f73c7a26544bd15738e937b65f" Nov 02 22:15:20.447616 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:20.447547 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477"] Nov 02 22:15:20.493574 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:20.493549 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-74477"] Nov 02 22:15:22.152156 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:22.152120 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" path="/var/lib/kubelet/pods/dc88dae8-c9b6-45ba-b8e3-a08feacca859/volumes" Nov 02 22:15:36.901480 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:36.901443 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:15:45.532861 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:15:45.532821 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:15:57.132341 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:15:57.132306 4510 scope.go:122] "RemoveContainer" containerID="6a59f75e0c82a6bcaee50d19b547a201475605e2ad7185c3daff0b0c82ce79ee" Nov 02 22:16:15.529842 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:16:15.529806 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:16:20.596567 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:16:20.596530 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:16:43.620018 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:16:43.619937 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:16:45.615566 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:16:45.615490 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:16:54.304940 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:16:54.304909 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:16:57.165739 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:16:57.165706 4510 scope.go:122] "RemoveContainer" containerID="554f979af37d2217f1693beaf90e73ab5054fb2ee56bcad1472e67e93fc9a7a1" Nov 02 22:17:05.042029 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:05.041996 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2"] Nov 02 22:17:05.087815 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:05.087776 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-hf2k2"] Nov 02 22:17:06.152696 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:06.152662 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19821182-4a56-4e53-b19a-1e401726e83c" path="/var/lib/kubelet/pods/19821182-4a56-4e53-b19a-1e401726e83c/volumes" Nov 02 22:17:15.529585 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:17:15.529548 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:17:25.821989 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.821955 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw"] Nov 02 22:17:25.822485 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.822448 4510 cpu_manager.go:414] "containerMap: removing container" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:17:25.822485 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.822463 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:17:25.822485 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.822473 4510 cpu_manager.go:414] "containerMap: removing container" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:17:25.822485 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.822478 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:17:25.822654 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.822587 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:17:25.822654 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.822595 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:17:25.826012 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.825990 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.892813 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.892782 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.892943 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.892817 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkx7x\" (UniqueName: \"kubernetes.io/projected/69d9a062-196a-4374-84a1-58aafb616e95-kube-api-access-gkx7x\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.892943 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.892856 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.893109 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.892995 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9a062-196a-4374-84a1-58aafb616e95-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.923576 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.923552 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:17:25.923750 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.923551 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d16d3a-serving-cert\"" Nov 02 22:17:25.923750 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.923619 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47p47\"" Nov 02 22:17:25.925438 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.925418 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw"] Nov 02 22:17:25.993804 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.993763 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.993954 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.993814 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkx7x\" (UniqueName: \"kubernetes.io/projected/69d9a062-196a-4374-84a1-58aafb616e95-kube-api-access-gkx7x\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.993954 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.993858 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.994056 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.993965 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9a062-196a-4374-84a1-58aafb616e95-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.994056 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:17:25.994033 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-serving-cert: secret "maas-api-e2e-worker-single-main-d16d3a-serving-cert" not found Nov 02 22:17:25.994163 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:17:25.994115 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls podName:69d9a062-196a-4374-84a1-58aafb616e95 nodeName:}" failed. No retries permitted until 2026-11-02 22:17:26.494093319 +0000 UTC m=+3872.470391089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls") pod "maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" (UID: "69d9a062-196a-4374-84a1-58aafb616e95") : secret "maas-api-e2e-worker-single-main-d16d3a-serving-cert" not found Nov 02 22:17:25.994637 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.994616 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9a062-196a-4374-84a1-58aafb616e95-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:25.996165 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:25.996141 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:26.049913 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:26.049894 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkx7x\" (UniqueName: \"kubernetes.io/projected/69d9a062-196a-4374-84a1-58aafb616e95-kube-api-access-gkx7x\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:26.499377 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:26.499342 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:26.501963 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:26.501933 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:26.739330 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:26.739292 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:27.114610 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:27.114583 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw"] Nov 02 22:17:27.122580 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:17:27.122555 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d9a062_196a_4374_84a1_58aafb616e95.slice/crio-12f85c047fe369fb175b3dc05c8923b26be2f8f01dc4e0416936cd3e18f66869 WatchSource:0}: Error finding container 12f85c047fe369fb175b3dc05c8923b26be2f8f01dc4e0416936cd3e18f66869: Status 404 returned error can't find the container with id 12f85c047fe369fb175b3dc05c8923b26be2f8f01dc4e0416936cd3e18f66869 Nov 02 22:17:27.124053 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:27.124034 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:17:27.976583 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:27.976551 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" event={"ID":"69d9a062-196a-4374-84a1-58aafb616e95","Type":"ContainerStarted","Data":"12f85c047fe369fb175b3dc05c8923b26be2f8f01dc4e0416936cd3e18f66869"} Nov 02 22:17:28.986705 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:28.986665 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" event={"ID":"69d9a062-196a-4374-84a1-58aafb616e95","Type":"ContainerStarted","Data":"b8bb772c2cba05fb330ee8ce7fac93e9902843cb9ad57f30b0769ac0ee4640ef"} Nov 02 22:17:28.988273 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:28.988246 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:29.097844 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:29.097795 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" podStartSLOduration=3.258143117 podStartE2EDuration="4.097775819s" totalImagesPullingTime="839.632702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:29.097286521 +0000 UTC m=+3875.073584296" watchObservedRunningTime="2026-11-02 22:17:29.097775819 +0000 UTC m=+3875.074073591" Nov 02 22:17:34.128428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:34.128391 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:17:36.003570 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:36.003532 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:17:45.534990 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:17:45.534955 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:17:53.105041 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:17:53.105002 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:18:13.489345 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:13.489255 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:18:15.533384 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:18:15.533349 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:18:22.043361 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:22.043326 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb"] Nov 02 22:18:22.052027 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:22.052002 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-l96qb"] Nov 02 22:18:22.153317 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:22.153283 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f609a3b6-ee87-42cb-9d9b-1a29296b4103" path="/var/lib/kubelet/pods/f609a3b6-ee87-42cb-9d9b-1a29296b4103/volumes" Nov 02 22:18:42.586898 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:42.586854 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:18:45.533172 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:18:45.533130 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:18:57.193806 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:57.193767 4510 scope.go:122] "RemoveContainer" containerID="27abf4f064d6fbe079dfdefa4277b36f9f90bae75ede3c5b2108aea3b5e966ab" Nov 02 22:18:57.313985 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:57.313958 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw"] Nov 02 22:18:57.314162 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:57.314142 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" podUID="69d9a062-196a-4374-84a1-58aafb616e95" containerName="maas-api" containerID="cri-o://b8bb772c2cba05fb330ee8ce7fac93e9902843cb9ad57f30b0769ac0ee4640ef" gracePeriod=30 Nov 02 22:18:58.433411 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.433381 4510 generic.go:356] "Generic (PLEG): container finished" podID="69d9a062-196a-4374-84a1-58aafb616e95" containerID="b8bb772c2cba05fb330ee8ce7fac93e9902843cb9ad57f30b0769ac0ee4640ef" exitCode=0 Nov 02 22:18:58.433734 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.433432 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" event={"ID":"69d9a062-196a-4374-84a1-58aafb616e95","Type":"ContainerDied","Data":"b8bb772c2cba05fb330ee8ce7fac93e9902843cb9ad57f30b0769ac0ee4640ef"} Nov 02 22:18:58.565156 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.565130 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:18:58.634877 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.634851 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d9a062-196a-4374-84a1-58aafb616e95-kube-api-access-gkx7x\" (UniqueName: \"kubernetes.io/projected/69d9a062-196a-4374-84a1-58aafb616e95-kube-api-access-gkx7x\") pod \"69d9a062-196a-4374-84a1-58aafb616e95\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " Nov 02 22:18:58.634976 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.634902 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-metrics-certs\") pod \"69d9a062-196a-4374-84a1-58aafb616e95\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " Nov 02 22:18:58.635062 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.635046 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d9a062-196a-4374-84a1-58aafb616e95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9a062-196a-4374-84a1-58aafb616e95-openshift-service-ca\") pod \"69d9a062-196a-4374-84a1-58aafb616e95\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " Nov 02 22:18:58.635150 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.635135 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls\") pod \"69d9a062-196a-4374-84a1-58aafb616e95\" (UID: \"69d9a062-196a-4374-84a1-58aafb616e95\") " Nov 02 22:18:58.635435 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.635406 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d9a062-196a-4374-84a1-58aafb616e95-openshift-service-ca" pod "69d9a062-196a-4374-84a1-58aafb616e95" (UID: "69d9a062-196a-4374-84a1-58aafb616e95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:58.636999 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.636972 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-metrics-certs" pod "69d9a062-196a-4374-84a1-58aafb616e95" (UID: "69d9a062-196a-4374-84a1-58aafb616e95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:58.637075 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.637001 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls" pod "69d9a062-196a-4374-84a1-58aafb616e95" (UID: "69d9a062-196a-4374-84a1-58aafb616e95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:58.637075 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.637054 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d9a062-196a-4374-84a1-58aafb616e95-kube-api-access-gkx7x" pod "69d9a062-196a-4374-84a1-58aafb616e95" (UID: "69d9a062-196a-4374-84a1-58aafb616e95"). InnerVolumeSpecName "kube-api-access-gkx7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:58.736122 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.736093 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d9a062-196a-4374-84a1-58aafb616e95-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:58.736122 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.736121 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:58.736294 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.736135 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkx7x\" (UniqueName: \"kubernetes.io/projected/69d9a062-196a-4374-84a1-58aafb616e95-kube-api-access-gkx7x\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:58.736294 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:58.736146 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d9a062-196a-4374-84a1-58aafb616e95-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:59.441391 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:59.441359 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" Nov 02 22:18:59.441391 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:59.441385 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw" event={"ID":"69d9a062-196a-4374-84a1-58aafb616e95","Type":"ContainerDied","Data":"12f85c047fe369fb175b3dc05c8923b26be2f8f01dc4e0416936cd3e18f66869"} Nov 02 22:18:59.441919 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:59.441446 4510 scope.go:122] "RemoveContainer" containerID="b8bb772c2cba05fb330ee8ce7fac93e9902843cb9ad57f30b0769ac0ee4640ef" Nov 02 22:18:59.591874 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:59.591845 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw"] Nov 02 22:18:59.638034 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:18:59.638004 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d16d3a-7685544899-vtwqw"] Nov 02 22:19:00.153466 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:00.153430 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d9a062-196a-4374-84a1-58aafb616e95" path="/var/lib/kubelet/pods/69d9a062-196a-4374-84a1-58aafb616e95/volumes" Nov 02 22:19:02.228454 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:02.228418 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:19:07.732366 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.732334 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7"] Nov 02 22:19:07.732890 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.732786 4510 cpu_manager.go:414] "containerMap: removing container" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:19:07.732890 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.732797 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:19:07.732890 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.732820 4510 cpu_manager.go:414] "containerMap: removing container" podUID="69d9a062-196a-4374-84a1-58aafb616e95" containerName="maas-api" Nov 02 22:19:07.732890 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.732826 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d9a062-196a-4374-84a1-58aafb616e95" containerName="maas-api" Nov 02 22:19:07.732890 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.732889 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc88dae8-c9b6-45ba-b8e3-a08feacca859" containerName="cleanup" Nov 02 22:19:07.733078 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.732901 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d9a062-196a-4374-84a1-58aafb616e95" containerName="maas-api" Nov 02 22:19:07.736146 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.736121 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.786724 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.786682 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:07.829873 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.829844 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.830006 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.829877 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.830006 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.829970 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqqk\" (UniqueName: \"kubernetes.io/projected/073f55b7-87d3-4793-bd03-f5e135869e2e-kube-api-access-xwqqk\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.832852 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.832837 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:19:07.834934 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.834914 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7"] Nov 02 22:19:07.931422 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.931383 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.931620 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.931429 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.931620 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.931533 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqqk\" (UniqueName: \"kubernetes.io/projected/073f55b7-87d3-4793-bd03-f5e135869e2e-kube-api-access-xwqqk\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.932104 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.932085 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.933664 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.933640 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:07.983855 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:07.983798 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqqk\" (UniqueName: \"kubernetes.io/projected/073f55b7-87d3-4793-bd03-f5e135869e2e-kube-api-access-xwqqk\") pod \"maas-api-revoke-sub-3d4156f1-d7bt7\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:08.049724 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:08.049700 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:08.416735 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:08.416703 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7"] Nov 02 22:19:08.424872 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:19:08.424843 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073f55b7_87d3_4793_bd03_f5e135869e2e.slice/crio-45e5e37f5534fe7f564df22c343ef75eed719a9c780f54b3ad768f530966bae3 WatchSource:0}: Error finding container 45e5e37f5534fe7f564df22c343ef75eed719a9c780f54b3ad768f530966bae3: Status 404 returned error can't find the container with id 45e5e37f5534fe7f564df22c343ef75eed719a9c780f54b3ad768f530966bae3 Nov 02 22:19:08.486565 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:08.486536 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" event={"ID":"073f55b7-87d3-4793-bd03-f5e135869e2e","Type":"ContainerStarted","Data":"45e5e37f5534fe7f564df22c343ef75eed719a9c780f54b3ad768f530966bae3"} Nov 02 22:19:09.493993 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:09.493904 4510 generic.go:356] "Generic (PLEG): container finished" podID="073f55b7-87d3-4793-bd03-f5e135869e2e" containerID="8c21c06d0b7d9fb499a2f52bd325a818b57a7eb64c651458dfa8b1a9ab60ba6b" exitCode=0 Nov 02 22:19:09.494354 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:09.493985 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" event={"ID":"073f55b7-87d3-4793-bd03-f5e135869e2e","Type":"ContainerDied","Data":"8c21c06d0b7d9fb499a2f52bd325a818b57a7eb64c651458dfa8b1a9ab60ba6b"} Nov 02 22:19:10.635318 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.635293 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:10.763365 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.763281 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-xks-ca\") pod \"073f55b7-87d3-4793-bd03-f5e135869e2e\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " Nov 02 22:19:10.763365 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.763352 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073f55b7-87d3-4793-bd03-f5e135869e2e-kube-api-access-xwqqk\" (UniqueName: \"kubernetes.io/projected/073f55b7-87d3-4793-bd03-f5e135869e2e-kube-api-access-xwqqk\") pod \"073f55b7-87d3-4793-bd03-f5e135869e2e\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " Nov 02 22:19:10.763630 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.763414 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-openshift-ca\") pod \"073f55b7-87d3-4793-bd03-f5e135869e2e\" (UID: \"073f55b7-87d3-4793-bd03-f5e135869e2e\") " Nov 02 22:19:10.763847 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.763818 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-openshift-ca" pod "073f55b7-87d3-4793-bd03-f5e135869e2e" (UID: "073f55b7-87d3-4793-bd03-f5e135869e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:10.765730 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.765705 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-xks-ca" pod "073f55b7-87d3-4793-bd03-f5e135869e2e" (UID: "073f55b7-87d3-4793-bd03-f5e135869e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:10.765881 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.765861 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073f55b7-87d3-4793-bd03-f5e135869e2e-kube-api-access-xwqqk" pod "073f55b7-87d3-4793-bd03-f5e135869e2e" (UID: "073f55b7-87d3-4793-bd03-f5e135869e2e"). InnerVolumeSpecName "kube-api-access-xwqqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:10.864490 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.864456 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-xks-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:10.864490 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.864488 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqqk\" (UniqueName: \"kubernetes.io/projected/073f55b7-87d3-4793-bd03-f5e135869e2e-kube-api-access-xwqqk\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:10.864706 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:10.864501 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/073f55b7-87d3-4793-bd03-f5e135869e2e-maas-api-openshift-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.506007 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:11.505979 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" Nov 02 22:19:11.506174 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:11.505972 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7" event={"ID":"073f55b7-87d3-4793-bd03-f5e135869e2e","Type":"ContainerDied","Data":"45e5e37f5534fe7f564df22c343ef75eed719a9c780f54b3ad768f530966bae3"} Nov 02 22:19:11.506174 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:11.506112 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e5e37f5534fe7f564df22c343ef75eed719a9c780f54b3ad768f530966bae3" Nov 02 22:19:15.532000 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:19:15.531962 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:19:45.534578 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:19:45.534469 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:19:57.213688 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:19:57.213656 4510 scope.go:122] "RemoveContainer" containerID="791419040df6dfa95d94b98aa6dab9a499ab985fe78291eb21f6486642c5e492" Nov 02 22:20:15.529877 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:20:15.529843 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:20:45.531660 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:20:45.531618 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:20:54.182546 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:20:54.182502 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:21:11.480007 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:11.479923 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:21:15.530148 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:21:15.530110 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:21:20.789024 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:20.788993 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:21:40.210956 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:40.210922 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:21:45.530982 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:21:45.530945 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:21:50.832566 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.832534 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp"] Nov 02 22:21:50.833004 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.832970 4510 cpu_manager.go:414] "containerMap: removing container" podUID="073f55b7-87d3-4793-bd03-f5e135869e2e" containerName="revoke-keys" Nov 02 22:21:50.833004 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.832980 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073f55b7-87d3-4793-bd03-f5e135869e2e" containerName="revoke-keys" Nov 02 22:21:50.833091 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.833064 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="073f55b7-87d3-4793-bd03-f5e135869e2e" containerName="revoke-keys" Nov 02 22:21:50.836437 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.836419 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:50.884956 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.884933 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c19d19-serving-cert\"" Nov 02 22:21:50.931218 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.931180 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:21:50.931353 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.931331 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47p47\"" Nov 02 22:21:50.933351 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:50.933331 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp"] Nov 02 22:21:51.006098 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.006068 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.006264 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.006112 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cwb\" (UniqueName: \"kubernetes.io/projected/ec2d8599-6846-471e-a789-89d0ca8d857f-kube-api-access-47cwb\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.006264 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.006212 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.006264 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.006246 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d8599-6846-471e-a789-89d0ca8d857f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.107514 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.107428 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d8599-6846-471e-a789-89d0ca8d857f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.107708 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.107597 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.107708 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.107646 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cwb\" (UniqueName: \"kubernetes.io/projected/ec2d8599-6846-471e-a789-89d0ca8d857f-kube-api-access-47cwb\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.107836 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.107705 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.108266 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.108242 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d8599-6846-471e-a789-89d0ca8d857f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.110306 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.110283 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.110412 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.110355 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.160160 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.160135 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cwb\" (UniqueName: \"kubernetes.io/projected/ec2d8599-6846-471e-a789-89d0ca8d857f-kube-api-access-47cwb\") pod \"maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.449282 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.449250 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:51.822013 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:51.821931 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp"] Nov 02 22:21:51.829489 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:21:51.829463 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d8599_6846_471e_a789_89d0ca8d857f.slice/crio-52cd12c77c9a150bf33e9ca03ceb425e8cdb419bd293e3038f6b40142a423cf8 WatchSource:0}: Error finding container 52cd12c77c9a150bf33e9ca03ceb425e8cdb419bd293e3038f6b40142a423cf8: Status 404 returned error can't find the container with id 52cd12c77c9a150bf33e9ca03ceb425e8cdb419bd293e3038f6b40142a423cf8 Nov 02 22:21:52.363835 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:52.363803 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" event={"ID":"ec2d8599-6846-471e-a789-89d0ca8d857f","Type":"ContainerStarted","Data":"52cd12c77c9a150bf33e9ca03ceb425e8cdb419bd293e3038f6b40142a423cf8"} Nov 02 22:21:53.370045 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:53.370006 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" event={"ID":"ec2d8599-6846-471e-a789-89d0ca8d857f","Type":"ContainerStarted","Data":"e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f"} Nov 02 22:21:53.370499 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:53.370137 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:21:53.477550 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:53.477488 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" podStartSLOduration=2.610092314 podStartE2EDuration="3.477472753s" totalImagesPullingTime="867.380439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:53.475792231 +0000 UTC m=+4139.452090017" watchObservedRunningTime="2026-11-02 22:21:53.477472753 +0000 UTC m=+4139.453770519" Nov 02 22:21:58.520034 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:58.520000 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:21:59.380894 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:21:59.380862 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:22:15.529904 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:22:15.529862 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:22:17.901470 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:22:17.901432 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:22:41.411701 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:22:41.411606 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:22:45.532419 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:22:45.532384 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:23:09.125851 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:09.125817 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:23:15.529781 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:23:15.529741 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:23:22.201265 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:22.201232 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp"] Nov 02 22:23:22.201851 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:22.201436 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" podUID="ec2d8599-6846-471e-a789-89d0ca8d857f" containerName="maas-api" containerID="cri-o://e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f" gracePeriod=30 Nov 02 22:23:23.452073 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.452050 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:23:23.527245 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.527180 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2d8599-6846-471e-a789-89d0ca8d857f-kube-api-access-47cwb\" (UniqueName: \"kubernetes.io/projected/ec2d8599-6846-471e-a789-89d0ca8d857f-kube-api-access-47cwb\") pod \"ec2d8599-6846-471e-a789-89d0ca8d857f\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " Nov 02 22:23:23.527245 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.527235 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2d8599-6846-471e-a789-89d0ca8d857f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d8599-6846-471e-a789-89d0ca8d857f-openshift-service-ca\") pod \"ec2d8599-6846-471e-a789-89d0ca8d857f\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " Nov 02 22:23:23.527452 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.527268 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-metrics-certs\") pod \"ec2d8599-6846-471e-a789-89d0ca8d857f\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " Nov 02 22:23:23.527452 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.527313 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-maas-api-tls\") pod \"ec2d8599-6846-471e-a789-89d0ca8d857f\" (UID: \"ec2d8599-6846-471e-a789-89d0ca8d857f\") " Nov 02 22:23:23.527695 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.527664 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2d8599-6846-471e-a789-89d0ca8d857f-openshift-service-ca" pod "ec2d8599-6846-471e-a789-89d0ca8d857f" (UID: "ec2d8599-6846-471e-a789-89d0ca8d857f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:23.527826 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.527755 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec2d8599-6846-471e-a789-89d0ca8d857f-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:23.529360 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.529335 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-metrics-certs" pod "ec2d8599-6846-471e-a789-89d0ca8d857f" (UID: "ec2d8599-6846-471e-a789-89d0ca8d857f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:23:23.529453 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.529377 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-maas-api-tls" pod "ec2d8599-6846-471e-a789-89d0ca8d857f" (UID: "ec2d8599-6846-471e-a789-89d0ca8d857f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:23:23.529453 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.529411 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2d8599-6846-471e-a789-89d0ca8d857f-kube-api-access-47cwb" pod "ec2d8599-6846-471e-a789-89d0ca8d857f" (UID: "ec2d8599-6846-471e-a789-89d0ca8d857f"). InnerVolumeSpecName "kube-api-access-47cwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:23.628364 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.628332 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:23.628364 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.628357 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec2d8599-6846-471e-a789-89d0ca8d857f-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:23.628364 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.628367 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cwb\" (UniqueName: \"kubernetes.io/projected/ec2d8599-6846-471e-a789-89d0ca8d857f-kube-api-access-47cwb\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:23.816560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.816469 4510 generic.go:356] "Generic (PLEG): container finished" podID="ec2d8599-6846-471e-a789-89d0ca8d857f" containerID="e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f" exitCode=0 Nov 02 22:23:23.816560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.816514 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" Nov 02 22:23:23.816560 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.816544 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" event={"ID":"ec2d8599-6846-471e-a789-89d0ca8d857f","Type":"ContainerDied","Data":"e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f"} Nov 02 22:23:23.816821 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.816585 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp" event={"ID":"ec2d8599-6846-471e-a789-89d0ca8d857f","Type":"ContainerDied","Data":"52cd12c77c9a150bf33e9ca03ceb425e8cdb419bd293e3038f6b40142a423cf8"} Nov 02 22:23:23.816821 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.816606 4510 scope.go:122] "RemoveContainer" containerID="e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f" Nov 02 22:23:23.826634 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.826616 4510 scope.go:122] "RemoveContainer" containerID="e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f" Nov 02 22:23:23.826914 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:23:23.826893 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f\": container with ID starting with e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f not found: ID does not exist" containerID="e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f" Nov 02 22:23:23.826959 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.826922 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f"} err="failed to get container status \"e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f\": rpc error: code = NotFound desc = could not find container \"e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f\": container with ID starting with e0aec7d04ed70f3942d00ec256a26bdff9ffbf0510d083652931bd6167c3c72f not found: ID does not exist" Nov 02 22:23:23.967842 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:23.967808 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp"] Nov 02 22:23:24.013875 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:24.013841 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c19d19-77f9476d97-ldjxp"] Nov 02 22:23:24.152700 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:24.152675 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2d8599-6846-471e-a789-89d0ca8d857f" path="/var/lib/kubelet/pods/ec2d8599-6846-471e-a789-89d0ca8d857f/volumes" Nov 02 22:23:26.797695 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:26.797656 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:23:41.678209 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:23:41.678175 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:23:45.536642 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:23:45.536603 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:24:11.042428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:11.042344 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7"] Nov 02 22:24:11.088324 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:11.088295 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d7bt7"] Nov 02 22:24:12.153005 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:12.152973 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073f55b7-87d3-4793-bd03-f5e135869e2e" path="/var/lib/kubelet/pods/073f55b7-87d3-4793-bd03-f5e135869e2e/volumes" Nov 02 22:24:15.530750 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:24:15.530714 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:24:23.313601 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:23.313571 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:24:26.101681 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:26.101612 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:24:26.102344 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:26.102292 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-kqfwb" podUID="63cbd378-8dec-4cb9-9a52-f145dec09563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:24:45.531051 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:24:45.531008 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:24:48.482193 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.482159 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt"] Nov 02 22:24:48.482657 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.482643 4510 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d8599-6846-471e-a789-89d0ca8d857f" containerName="maas-api" Nov 02 22:24:48.482713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.482658 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d8599-6846-471e-a789-89d0ca8d857f" containerName="maas-api" Nov 02 22:24:48.482775 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.482766 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d8599-6846-471e-a789-89d0ca8d857f" containerName="maas-api" Nov 02 22:24:48.486059 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.486036 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.536714 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.536695 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:48.582555 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.582532 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:24:48.584769 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.584746 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt"] Nov 02 22:24:48.623138 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.623102 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.623295 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.623265 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksbx\" (UniqueName: \"kubernetes.io/projected/91b7be79-9b36-491b-81ca-7093abaf1351-kube-api-access-cksbx\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.623375 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.623350 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.724190 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.724161 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksbx\" (UniqueName: \"kubernetes.io/projected/91b7be79-9b36-491b-81ca-7093abaf1351-kube-api-access-cksbx\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.724333 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.724193 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.724333 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.724273 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.724869 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.724847 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.726590 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.726569 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.776974 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.776914 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksbx\" (UniqueName: \"kubernetes.io/projected/91b7be79-9b36-491b-81ca-7093abaf1351-kube-api-access-cksbx\") pod \"maas-api-revoke-sub-452b6d7b-4cxrt\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:48.799062 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:48.799039 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:49.165762 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:49.165726 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt"] Nov 02 22:24:49.173973 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:24:49.173946 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b7be79_9b36_491b_81ca_7093abaf1351.slice/crio-62023d844eed9f84d941e85406e9497be59610201b7061cb793ea72215c74e01 WatchSource:0}: Error finding container 62023d844eed9f84d941e85406e9497be59610201b7061cb793ea72215c74e01: Status 404 returned error can't find the container with id 62023d844eed9f84d941e85406e9497be59610201b7061cb793ea72215c74e01 Nov 02 22:24:49.254888 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:49.254848 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" event={"ID":"91b7be79-9b36-491b-81ca-7093abaf1351","Type":"ContainerStarted","Data":"62023d844eed9f84d941e85406e9497be59610201b7061cb793ea72215c74e01"} Nov 02 22:24:49.722804 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:49.722711 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:24:50.263453 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:50.263357 4510 generic.go:356] "Generic (PLEG): container finished" podID="91b7be79-9b36-491b-81ca-7093abaf1351" containerID="f8e4dcfe423478a846a2ba94301648e21346e169efbbbccb68a05e7a48b2437d" exitCode=0 Nov 02 22:24:50.263453 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:50.263445 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" event={"ID":"91b7be79-9b36-491b-81ca-7093abaf1351","Type":"ContainerDied","Data":"f8e4dcfe423478a846a2ba94301648e21346e169efbbbccb68a05e7a48b2437d"} Nov 02 22:24:51.614867 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.614843 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:24:51.802146 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.802113 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-openshift-ca\") pod \"91b7be79-9b36-491b-81ca-7093abaf1351\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " Nov 02 22:24:51.802433 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.802410 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-xks-ca\") pod \"91b7be79-9b36-491b-81ca-7093abaf1351\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " Nov 02 22:24:51.802786 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.802762 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b7be79-9b36-491b-81ca-7093abaf1351-kube-api-access-cksbx\" (UniqueName: \"kubernetes.io/projected/91b7be79-9b36-491b-81ca-7093abaf1351-kube-api-access-cksbx\") pod \"91b7be79-9b36-491b-81ca-7093abaf1351\" (UID: \"91b7be79-9b36-491b-81ca-7093abaf1351\") " Nov 02 22:24:51.804431 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.804144 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-openshift-ca" pod "91b7be79-9b36-491b-81ca-7093abaf1351" (UID: "91b7be79-9b36-491b-81ca-7093abaf1351"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:51.818994 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.818965 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b7be79-9b36-491b-81ca-7093abaf1351-kube-api-access-cksbx" pod "91b7be79-9b36-491b-81ca-7093abaf1351" (UID: "91b7be79-9b36-491b-81ca-7093abaf1351"). InnerVolumeSpecName "kube-api-access-cksbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:51.820622 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.820595 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-xks-ca" pod "91b7be79-9b36-491b-81ca-7093abaf1351" (UID: "91b7be79-9b36-491b-81ca-7093abaf1351"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:51.904811 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.904757 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksbx\" (UniqueName: \"kubernetes.io/projected/91b7be79-9b36-491b-81ca-7093abaf1351-kube-api-access-cksbx\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:51.905009 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.904814 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-openshift-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:51.905009 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:51.904832 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b7be79-9b36-491b-81ca-7093abaf1351-maas-api-xks-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:52.296584 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:52.296476 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" event={"ID":"91b7be79-9b36-491b-81ca-7093abaf1351","Type":"ContainerDied","Data":"62023d844eed9f84d941e85406e9497be59610201b7061cb793ea72215c74e01"} Nov 02 22:24:52.296584 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:52.296573 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62023d844eed9f84d941e85406e9497be59610201b7061cb793ea72215c74e01" Nov 02 22:24:52.296854 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:24:52.296675 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt" Nov 02 22:25:04.831883 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:04.831849 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:25:14.155717 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.155675 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t"] Nov 02 22:25:14.156428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.156407 4510 cpu_manager.go:414] "containerMap: removing container" podUID="91b7be79-9b36-491b-81ca-7093abaf1351" containerName="revoke-keys" Nov 02 22:25:14.156567 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.156433 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b7be79-9b36-491b-81ca-7093abaf1351" containerName="revoke-keys" Nov 02 22:25:14.156662 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.156648 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b7be79-9b36-491b-81ca-7093abaf1351" containerName="revoke-keys" Nov 02 22:25:14.160741 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.160718 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.203999 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.203073 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t"] Nov 02 22:25:14.245621 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.245591 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:14.246011 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.245975 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j922b\"" Nov 02 22:25:14.266326 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.266296 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.266461 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.266358 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcs5b\" (UniqueName: \"kubernetes.io/projected/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-kube-api-access-mcs5b\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.266461 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.266407 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.368110 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.368063 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.368388 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.368365 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcs5b\" (UniqueName: \"kubernetes.io/projected/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-kube-api-access-mcs5b\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.368547 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.368509 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.369619 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.369595 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.385222 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.385193 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.423306 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.423230 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcs5b\" (UniqueName: \"kubernetes.io/projected/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-kube-api-access-mcs5b\") pod \"maas-api-revoke-sub-3e31a506-xhc6t\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.490543 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.489716 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:14.962543 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:14.962169 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t"] Nov 02 22:25:15.495543 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:15.494431 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" event={"ID":"82165f85-9b80-4522-b1e8-4d13f3b6e6cb","Type":"ContainerStarted","Data":"67b44b8a29d2828c8c9f6ac13cfe277bd27717bc636f851390683ecfd8ec6c4d"} Nov 02 22:25:15.581913 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:25:15.581759 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:25:16.501344 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:16.501257 4510 generic.go:356] "Generic (PLEG): container finished" podID="82165f85-9b80-4522-b1e8-4d13f3b6e6cb" containerID="f9bda099bd8605e3bddfb3f56d8f6d1bec3015266c1a6feb99e1ab10e8fad968" exitCode=0 Nov 02 22:25:16.501745 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:16.501352 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" event={"ID":"82165f85-9b80-4522-b1e8-4d13f3b6e6cb","Type":"ContainerDied","Data":"f9bda099bd8605e3bddfb3f56d8f6d1bec3015266c1a6feb99e1ab10e8fad968"} Nov 02 22:25:17.633046 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.633021 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:17.739839 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.739808 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-kube-api-access-mcs5b\" (UniqueName: \"kubernetes.io/projected/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-kube-api-access-mcs5b\") pod \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " Nov 02 22:25:17.740044 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.739863 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-openshift-ca\") pod \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " Nov 02 22:25:17.740044 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.739937 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-xks-ca\") pod \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\" (UID: \"82165f85-9b80-4522-b1e8-4d13f3b6e6cb\") " Nov 02 22:25:17.740332 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.740311 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-openshift-ca" pod "82165f85-9b80-4522-b1e8-4d13f3b6e6cb" (UID: "82165f85-9b80-4522-b1e8-4d13f3b6e6cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:17.742565 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.742542 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-kube-api-access-mcs5b" pod "82165f85-9b80-4522-b1e8-4d13f3b6e6cb" (UID: "82165f85-9b80-4522-b1e8-4d13f3b6e6cb"). InnerVolumeSpecName "kube-api-access-mcs5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:17.742658 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.742591 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-xks-ca" pod "82165f85-9b80-4522-b1e8-4d13f3b6e6cb" (UID: "82165f85-9b80-4522-b1e8-4d13f3b6e6cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:17.841089 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.841021 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-openshift-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:17.841089 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.841048 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-maas-api-xks-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:17.841089 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:17.841059 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcs5b\" (UniqueName: \"kubernetes.io/projected/82165f85-9b80-4522-b1e8-4d13f3b6e6cb-kube-api-access-mcs5b\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:18.512727 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:18.512638 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" event={"ID":"82165f85-9b80-4522-b1e8-4d13f3b6e6cb","Type":"ContainerDied","Data":"67b44b8a29d2828c8c9f6ac13cfe277bd27717bc636f851390683ecfd8ec6c4d"} Nov 02 22:25:18.512727 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:18.512668 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t" Nov 02 22:25:18.512727 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:18.512700 4510 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67b44b8a29d2828c8c9f6ac13cfe277bd27717bc636f851390683ecfd8ec6c4d" Nov 02 22:25:19.807763 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:19.807734 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:25:30.778779 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:30.778742 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:25:45.530992 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:25:45.530914 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:25:50.812743 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:50.812710 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:25:51.208214 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.208169 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97"] Nov 02 22:25:51.208969 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.208946 4510 cpu_manager.go:414] "containerMap: removing container" podUID="82165f85-9b80-4522-b1e8-4d13f3b6e6cb" containerName="revoke-keys" Nov 02 22:25:51.209055 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.208971 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82165f85-9b80-4522-b1e8-4d13f3b6e6cb" containerName="revoke-keys" Nov 02 22:25:51.209115 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.209101 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="82165f85-9b80-4522-b1e8-4d13f3b6e6cb" containerName="revoke-keys" Nov 02 22:25:51.213362 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.213229 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.220171 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.220146 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn"] Nov 02 22:25:51.223879 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.223858 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.264137 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.264108 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:25:51.264292 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.264188 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47p47\"" Nov 02 22:25:51.298819 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.298791 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69vl\" (UniqueName: \"kubernetes.io/projected/69176f08-999b-465b-ae1c-78e1c4ae9247-kube-api-access-k69vl\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.298945 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.298828 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-metrics-certs\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.298945 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.298852 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-openshift-service-ca\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.299040 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.298943 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-maas-api-tls\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.299040 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.299008 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69176f08-999b-465b-ae1c-78e1c4ae9247-openshift-service-ca\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.299119 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.299071 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-maas-api-tls\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.299119 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.299089 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq772\" (UniqueName: \"kubernetes.io/projected/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-kube-api-access-wq772\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.299119 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.299110 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-metrics-certs\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.308082 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.308060 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 22:25:51.310775 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.310755 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97"] Nov 02 22:25:51.364622 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.364596 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn"] Nov 02 22:25:51.399801 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.399776 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-metrics-certs\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.399940 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.399873 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69vl\" (UniqueName: \"kubernetes.io/projected/69176f08-999b-465b-ae1c-78e1c4ae9247-kube-api-access-k69vl\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.399940 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.399900 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-metrics-certs\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.399940 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.399924 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-openshift-service-ca\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.400050 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.399995 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-maas-api-tls\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.400050 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.400033 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69176f08-999b-465b-ae1c-78e1c4ae9247-openshift-service-ca\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.400135 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.400094 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-maas-api-tls\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.400135 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.400121 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq772\" (UniqueName: \"kubernetes.io/projected/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-kube-api-access-wq772\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.400736 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.400701 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-openshift-service-ca\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.400871 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.400813 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69176f08-999b-465b-ae1c-78e1c4ae9247-openshift-service-ca\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.402596 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.402568 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-metrics-certs\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.402720 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.402671 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-maas-api-tls\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.402841 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.402824 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-metrics-certs\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.402943 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.402921 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-maas-api-tls\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.453270 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.453239 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69vl\" (UniqueName: \"kubernetes.io/projected/69176f08-999b-465b-ae1c-78e1c4ae9247-kube-api-access-k69vl\") pod \"maas-api-5d98fc7974-ljk97\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.453385 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.453246 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq772\" (UniqueName: \"kubernetes.io/projected/a5a190ca-b18a-4c79-a3e9-92dfc23f19e2-kube-api-access-wq772\") pod \"maas-api-8455db45c4-n7fsn\" (UID: \"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2\") " pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.530310 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.530248 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:51.538314 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.538289 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:51.907259 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.907220 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97"] Nov 02 22:25:51.915745 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:25:51.915717 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69176f08_999b_465b_ae1c_78e1c4ae9247.slice/crio-3d5eabcf14e7011b864e3c98ceb69a1e1ceeddf39844c64ca997eeafe57c0581 WatchSource:0}: Error finding container 3d5eabcf14e7011b864e3c98ceb69a1e1ceeddf39844c64ca997eeafe57c0581: Status 404 returned error can't find the container with id 3d5eabcf14e7011b864e3c98ceb69a1e1ceeddf39844c64ca997eeafe57c0581 Nov 02 22:25:51.917150 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.917134 4510 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:25:51.955316 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:51.955283 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn"] Nov 02 22:25:51.962773 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:25:51.962725 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a190ca_b18a_4c79_a3e9_92dfc23f19e2.slice/crio-cabe72a37b657ed79f3035f1633852420406b75a72e9ca7cc9d498eef92c4047 WatchSource:0}: Error finding container cabe72a37b657ed79f3035f1633852420406b75a72e9ca7cc9d498eef92c4047: Status 404 returned error can't find the container with id cabe72a37b657ed79f3035f1633852420406b75a72e9ca7cc9d498eef92c4047 Nov 02 22:25:52.720041 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:52.720003 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" event={"ID":"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2","Type":"ContainerStarted","Data":"cabe72a37b657ed79f3035f1633852420406b75a72e9ca7cc9d498eef92c4047"} Nov 02 22:25:52.721223 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:52.721196 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" event={"ID":"69176f08-999b-465b-ae1c-78e1c4ae9247","Type":"ContainerStarted","Data":"3d5eabcf14e7011b864e3c98ceb69a1e1ceeddf39844c64ca997eeafe57c0581"} Nov 02 22:25:53.727491 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:53.727456 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" event={"ID":"69176f08-999b-465b-ae1c-78e1c4ae9247","Type":"ContainerStarted","Data":"599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890"} Nov 02 22:25:53.727965 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:53.727552 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:53.729067 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:53.729046 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" event={"ID":"a5a190ca-b18a-4c79-a3e9-92dfc23f19e2","Type":"ContainerStarted","Data":"0a3b78947bdbbc98c9d28df7d26e24e110f09e6cd223d76e2cbec0d2bff8d8d4"} Nov 02 22:25:53.729145 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:53.729122 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:53.833156 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:53.833108 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" podStartSLOduration=1.917301559 podStartE2EDuration="2.833094726s" totalImagesPullingTime="915.793167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:53.832471107 +0000 UTC m=+4379.808768881" watchObservedRunningTime="2026-11-02 22:25:53.833094726 +0000 UTC m=+4379.809392501" Nov 02 22:25:53.939014 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:53.938966 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" podStartSLOduration=2.0285455 podStartE2EDuration="2.938952302s" totalImagesPullingTime="910.406802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:53.938261508 +0000 UTC m=+4379.914559287" watchObservedRunningTime="2026-11-02 22:25:53.938952302 +0000 UTC m=+4379.915250077" Nov 02 22:25:57.277514 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:57.277484 4510 scope.go:122] "RemoveContainer" containerID="8c21c06d0b7d9fb499a2f52bd325a818b57a7eb64c651458dfa8b1a9ab60ba6b" Nov 02 22:25:59.738652 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:59.738624 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8455db45c4-n7fsn" Nov 02 22:25:59.739075 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:59.739020 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:25:59.893589 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:59.893559 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97"] Nov 02 22:25:59.893774 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:25:59.893736 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" podUID="69176f08-999b-465b-ae1c-78e1c4ae9247" containerName="maas-api" containerID="cri-o://599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890" gracePeriod=30 Nov 02 22:26:00.151924 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.151900 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:26:00.293409 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.293329 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-maas-api-tls\") pod \"69176f08-999b-465b-ae1c-78e1c4ae9247\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " Nov 02 22:26:00.293409 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.293395 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-metrics-certs\") pod \"69176f08-999b-465b-ae1c-78e1c4ae9247\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " Nov 02 22:26:00.293668 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.293428 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69176f08-999b-465b-ae1c-78e1c4ae9247-kube-api-access-k69vl\" (UniqueName: \"kubernetes.io/projected/69176f08-999b-465b-ae1c-78e1c4ae9247-kube-api-access-k69vl\") pod \"69176f08-999b-465b-ae1c-78e1c4ae9247\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " Nov 02 22:26:00.293668 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.293636 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69176f08-999b-465b-ae1c-78e1c4ae9247-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69176f08-999b-465b-ae1c-78e1c4ae9247-openshift-service-ca\") pod \"69176f08-999b-465b-ae1c-78e1c4ae9247\" (UID: \"69176f08-999b-465b-ae1c-78e1c4ae9247\") " Nov 02 22:26:00.294033 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.294005 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69176f08-999b-465b-ae1c-78e1c4ae9247-openshift-service-ca" pod "69176f08-999b-465b-ae1c-78e1c4ae9247" (UID: "69176f08-999b-465b-ae1c-78e1c4ae9247"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:00.295570 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.295532 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-maas-api-tls" pod "69176f08-999b-465b-ae1c-78e1c4ae9247" (UID: "69176f08-999b-465b-ae1c-78e1c4ae9247"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:00.295661 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.295605 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-metrics-certs" pod "69176f08-999b-465b-ae1c-78e1c4ae9247" (UID: "69176f08-999b-465b-ae1c-78e1c4ae9247"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:00.295743 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.295728 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69176f08-999b-465b-ae1c-78e1c4ae9247-kube-api-access-k69vl" pod "69176f08-999b-465b-ae1c-78e1c4ae9247" (UID: "69176f08-999b-465b-ae1c-78e1c4ae9247"). InnerVolumeSpecName "kube-api-access-k69vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:00.395126 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.395086 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:00.395126 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.395115 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69176f08-999b-465b-ae1c-78e1c4ae9247-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:00.395126 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.395125 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69vl\" (UniqueName: \"kubernetes.io/projected/69176f08-999b-465b-ae1c-78e1c4ae9247-kube-api-access-k69vl\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:00.395126 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.395135 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69176f08-999b-465b-ae1c-78e1c4ae9247-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:00.767274 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.767236 4510 generic.go:356] "Generic (PLEG): container finished" podID="69176f08-999b-465b-ae1c-78e1c4ae9247" containerID="599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890" exitCode=0 Nov 02 22:26:00.767825 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.767286 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" Nov 02 22:26:00.767825 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.767320 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" event={"ID":"69176f08-999b-465b-ae1c-78e1c4ae9247","Type":"ContainerDied","Data":"599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890"} Nov 02 22:26:00.767825 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.767388 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97" event={"ID":"69176f08-999b-465b-ae1c-78e1c4ae9247","Type":"ContainerDied","Data":"3d5eabcf14e7011b864e3c98ceb69a1e1ceeddf39844c64ca997eeafe57c0581"} Nov 02 22:26:00.767825 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.767413 4510 scope.go:122] "RemoveContainer" containerID="599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890" Nov 02 22:26:00.780421 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.780175 4510 scope.go:122] "RemoveContainer" containerID="599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890" Nov 02 22:26:00.780489 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:26:00.780439 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890\": container with ID starting with 599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890 not found: ID does not exist" containerID="599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890" Nov 02 22:26:00.780489 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.780471 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890"} err="failed to get container status \"599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890\": rpc error: code = NotFound desc = could not find container \"599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890\": container with ID starting with 599263a6a31575757a594f0c662d748a39824867fa14f70dde901cee634c6890 not found: ID does not exist" Nov 02 22:26:00.918978 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.918950 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97"] Nov 02 22:26:00.964996 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:00.964969 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d98fc7974-ljk97"] Nov 02 22:26:02.153304 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:02.153258 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69176f08-999b-465b-ae1c-78e1c4ae9247" path="/var/lib/kubelet/pods/69176f08-999b-465b-ae1c-78e1c4ae9247/volumes" Nov 02 22:26:05.826432 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.826398 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx"] Nov 02 22:26:05.826878 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.826871 4510 cpu_manager.go:414] "containerMap: removing container" podUID="69176f08-999b-465b-ae1c-78e1c4ae9247" containerName="maas-api" Nov 02 22:26:05.826932 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.826882 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69176f08-999b-465b-ae1c-78e1c4ae9247" containerName="maas-api" Nov 02 22:26:05.827014 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.827001 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="69176f08-999b-465b-ae1c-78e1c4ae9247" containerName="maas-api" Nov 02 22:26:05.830130 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.830109 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.852424 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.852400 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.852541 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.852461 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39ca3e4-41c1-4785-9f06-71a31b9012be-openshift-service-ca\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.852589 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.852558 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-metrics-certs\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.852627 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.852588 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsw5r\" (UniqueName: \"kubernetes.io/projected/a39ca3e4-41c1-4785-9f06-71a31b9012be-kube-api-access-qsw5r\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.878131 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.878111 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0abfb3fc-serving-cert\"" Nov 02 22:26:05.927710 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.927670 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx"] Nov 02 22:26:05.953862 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.953832 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsw5r\" (UniqueName: \"kubernetes.io/projected/a39ca3e4-41c1-4785-9f06-71a31b9012be-kube-api-access-qsw5r\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.954041 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.953969 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.954041 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.954010 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39ca3e4-41c1-4785-9f06-71a31b9012be-openshift-service-ca\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.954174 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.954051 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-metrics-certs\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.954174 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:26:05.954125 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-serving-cert: secret "maas-api-e2e-ait-0abfb3fc-serving-cert" not found Nov 02 22:26:05.954294 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:26:05.954231 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls podName:a39ca3e4-41c1-4785-9f06-71a31b9012be nodeName:}" failed. No retries permitted until 2026-11-02 22:26:06.454209392 +0000 UTC m=+4392.430507164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls") pod "maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" (UID: "a39ca3e4-41c1-4785-9f06-71a31b9012be") : secret "maas-api-e2e-ait-0abfb3fc-serving-cert" not found Nov 02 22:26:05.954761 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.954743 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39ca3e4-41c1-4785-9f06-71a31b9012be-openshift-service-ca\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:05.956659 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:05.956638 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-metrics-certs\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:06.006841 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:06.006809 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsw5r\" (UniqueName: \"kubernetes.io/projected/a39ca3e4-41c1-4785-9f06-71a31b9012be-kube-api-access-qsw5r\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:06.459504 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:06.459464 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:06.461958 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:06.461934 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls\") pod \"maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:06.743351 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:06.743257 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:07.112126 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:07.112032 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx"] Nov 02 22:26:07.119701 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:26:07.119672 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39ca3e4_41c1_4785_9f06_71a31b9012be.slice/crio-989cf9ae649b2cd0eaa0b3cb3cd62d6f4b79769bc07461c7147d106ebf36a13c WatchSource:0}: Error finding container 989cf9ae649b2cd0eaa0b3cb3cd62d6f4b79769bc07461c7147d106ebf36a13c: Status 404 returned error can't find the container with id 989cf9ae649b2cd0eaa0b3cb3cd62d6f4b79769bc07461c7147d106ebf36a13c Nov 02 22:26:07.813656 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:07.813623 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" event={"ID":"a39ca3e4-41c1-4785-9f06-71a31b9012be","Type":"ContainerStarted","Data":"989cf9ae649b2cd0eaa0b3cb3cd62d6f4b79769bc07461c7147d106ebf36a13c"} Nov 02 22:26:08.820925 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:08.820884 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" event={"ID":"a39ca3e4-41c1-4785-9f06-71a31b9012be","Type":"ContainerStarted","Data":"d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177"} Nov 02 22:26:08.821398 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:08.821196 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:08.928652 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:08.928603 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" podStartSLOduration=3.080553038 podStartE2EDuration="3.92858437s" totalImagesPullingTime="848.031332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:08.92680637 +0000 UTC m=+4394.903104145" watchObservedRunningTime="2026-11-02 22:26:08.92858437 +0000 UTC m=+4394.904882145" Nov 02 22:26:13.627333 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:13.627296 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:26:14.831365 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:14.831335 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:15.529124 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:26:15.529091 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:26:45.531549 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:26:45.531495 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:26:51.601363 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.601332 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx"] Nov 02 22:26:51.601870 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.601530 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" podUID="a39ca3e4-41c1-4785-9f06-71a31b9012be" containerName="maas-api" containerID="cri-o://d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177" gracePeriod=30 Nov 02 22:26:51.850506 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.850483 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:51.932066 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.932046 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-metrics-certs\") pod \"a39ca3e4-41c1-4785-9f06-71a31b9012be\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " Nov 02 22:26:51.932200 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.932077 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls\") pod \"a39ca3e4-41c1-4785-9f06-71a31b9012be\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " Nov 02 22:26:51.932200 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.932126 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39ca3e4-41c1-4785-9f06-71a31b9012be-kube-api-access-qsw5r\" (UniqueName: \"kubernetes.io/projected/a39ca3e4-41c1-4785-9f06-71a31b9012be-kube-api-access-qsw5r\") pod \"a39ca3e4-41c1-4785-9f06-71a31b9012be\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " Nov 02 22:26:51.932200 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.932179 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39ca3e4-41c1-4785-9f06-71a31b9012be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39ca3e4-41c1-4785-9f06-71a31b9012be-openshift-service-ca\") pod \"a39ca3e4-41c1-4785-9f06-71a31b9012be\" (UID: \"a39ca3e4-41c1-4785-9f06-71a31b9012be\") " Nov 02 22:26:51.932641 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.932617 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39ca3e4-41c1-4785-9f06-71a31b9012be-openshift-service-ca" pod "a39ca3e4-41c1-4785-9f06-71a31b9012be" (UID: "a39ca3e4-41c1-4785-9f06-71a31b9012be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:51.934205 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.934176 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls" pod "a39ca3e4-41c1-4785-9f06-71a31b9012be" (UID: "a39ca3e4-41c1-4785-9f06-71a31b9012be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:51.934205 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.934190 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-metrics-certs" pod "a39ca3e4-41c1-4785-9f06-71a31b9012be" (UID: "a39ca3e4-41c1-4785-9f06-71a31b9012be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:51.934335 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:51.934281 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39ca3e4-41c1-4785-9f06-71a31b9012be-kube-api-access-qsw5r" pod "a39ca3e4-41c1-4785-9f06-71a31b9012be" (UID: "a39ca3e4-41c1-4785-9f06-71a31b9012be"). InnerVolumeSpecName "kube-api-access-qsw5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:52.031997 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.031962 4510 generic.go:356] "Generic (PLEG): container finished" podID="a39ca3e4-41c1-4785-9f06-71a31b9012be" containerID="d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177" exitCode=0 Nov 02 22:26:52.032193 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.032012 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" Nov 02 22:26:52.032193 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.032060 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" event={"ID":"a39ca3e4-41c1-4785-9f06-71a31b9012be","Type":"ContainerDied","Data":"d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177"} Nov 02 22:26:52.032193 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.032128 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx" event={"ID":"a39ca3e4-41c1-4785-9f06-71a31b9012be","Type":"ContainerDied","Data":"989cf9ae649b2cd0eaa0b3cb3cd62d6f4b79769bc07461c7147d106ebf36a13c"} Nov 02 22:26:52.032193 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.032156 4510 scope.go:122] "RemoveContainer" containerID="d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177" Nov 02 22:26:52.033129 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.033105 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39ca3e4-41c1-4785-9f06-71a31b9012be-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.033129 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.033132 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.033324 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.033148 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39ca3e4-41c1-4785-9f06-71a31b9012be-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.033324 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.033158 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsw5r\" (UniqueName: \"kubernetes.io/projected/a39ca3e4-41c1-4785-9f06-71a31b9012be-kube-api-access-qsw5r\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.042395 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.042376 4510 scope.go:122] "RemoveContainer" containerID="d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177" Nov 02 22:26:52.042675 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:26:52.042658 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177\": container with ID starting with d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177 not found: ID does not exist" containerID="d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177" Nov 02 22:26:52.042719 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.042682 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177"} err="failed to get container status \"d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177\": rpc error: code = NotFound desc = could not find container \"d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177\": container with ID starting with d130986b6791699d9e682f3949bf7ae949bb1f04540b4b3dba59704946d7f177 not found: ID does not exist" Nov 02 22:26:52.183129 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.183049 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx"] Nov 02 22:26:52.186484 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:52.186458 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0abfb3fc-78f897d994-6whjx"] Nov 02 22:26:54.153251 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:54.153219 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39ca3e4-41c1-4785-9f06-71a31b9012be" path="/var/lib/kubelet/pods/a39ca3e4-41c1-4785-9f06-71a31b9012be/volumes" Nov 02 22:26:56.030907 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:26:56.030873 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:27:10.228724 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.228650 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4"] Nov 02 22:27:10.229117 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.229104 4510 cpu_manager.go:414] "containerMap: removing container" podUID="a39ca3e4-41c1-4785-9f06-71a31b9012be" containerName="maas-api" Nov 02 22:27:10.229170 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.229118 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39ca3e4-41c1-4785-9f06-71a31b9012be" containerName="maas-api" Nov 02 22:27:10.229215 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.229206 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39ca3e4-41c1-4785-9f06-71a31b9012be" containerName="maas-api" Nov 02 22:27:10.231873 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.231846 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.280151 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.280120 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-d9f155-serving-cert\"" Nov 02 22:27:10.312268 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.312230 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.312428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.312279 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjpx\" (UniqueName: \"kubernetes.io/projected/bd387dc4-be02-428d-90e5-73763b939199-kube-api-access-2jjpx\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.312428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.312373 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.312758 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.312438 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd387dc4-be02-428d-90e5-73763b939199-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.328562 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.328532 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4"] Nov 02 22:27:10.413742 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.413706 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.413957 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.413750 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjpx\" (UniqueName: \"kubernetes.io/projected/bd387dc4-be02-428d-90e5-73763b939199-kube-api-access-2jjpx\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.413957 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.413785 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.413957 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.413848 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd387dc4-be02-428d-90e5-73763b939199-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.413957 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:27:10.413858 4510 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-serving-cert: secret "maas-api-e2e-authcfg-http-d9f155-serving-cert" not found Nov 02 22:27:10.413957 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:27:10.413932 4510 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls podName:bd387dc4-be02-428d-90e5-73763b939199 nodeName:}" failed. No retries permitted until 2026-11-02 22:27:10.913911764 +0000 UTC m=+4456.890209517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls") pod "maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" (UID: "bd387dc4-be02-428d-90e5-73763b939199") : secret "maas-api-e2e-authcfg-http-d9f155-serving-cert" not found Nov 02 22:27:10.414568 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.414551 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd387dc4-be02-428d-90e5-73763b939199-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.416251 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.416232 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-metrics-certs\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.468159 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.468132 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjpx\" (UniqueName: \"kubernetes.io/projected/bd387dc4-be02-428d-90e5-73763b939199-kube-api-access-2jjpx\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.918714 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.918677 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:10.921110 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:10.921079 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:11.145089 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:11.145054 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:11.515985 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:11.515899 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4"] Nov 02 22:27:11.523760 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:27:11.523733 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd387dc4_be02_428d_90e5_73763b939199.slice/crio-2625b02d4f35af6aa71601393a501f9b9ff8525cdcfc70ad22c2873798c533e2 WatchSource:0}: Error finding container 2625b02d4f35af6aa71601393a501f9b9ff8525cdcfc70ad22c2873798c533e2: Status 404 returned error can't find the container with id 2625b02d4f35af6aa71601393a501f9b9ff8525cdcfc70ad22c2873798c533e2 Nov 02 22:27:12.135243 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:12.135209 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" event={"ID":"bd387dc4-be02-428d-90e5-73763b939199","Type":"ContainerStarted","Data":"2625b02d4f35af6aa71601393a501f9b9ff8525cdcfc70ad22c2873798c533e2"} Nov 02 22:27:13.142584 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:13.142545 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" event={"ID":"bd387dc4-be02-428d-90e5-73763b939199","Type":"ContainerStarted","Data":"ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b"} Nov 02 22:27:13.143059 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:13.142600 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:13.249085 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:13.249038 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" podStartSLOduration=2.228586859 podStartE2EDuration="3.249026066s" totalImagesPullingTime="1.020439207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:13.248027929 +0000 UTC m=+4459.224325704" watchObservedRunningTime="2026-11-02 22:27:13.249026066 +0000 UTC m=+4459.225323841" Nov 02 22:27:15.530347 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:27:15.530315 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:27:17.898076 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:17.898047 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:27:19.155492 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:27:19.155460 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:27:45.534501 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:27:45.534458 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:28:15.529979 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:28:15.529945 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:28:45.533723 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:28:45.533634 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:29:13.364085 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.364039 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4"] Nov 02 22:29:13.364658 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.364259 4510 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" podUID="bd387dc4-be02-428d-90e5-73763b939199" containerName="maas-api" containerID="cri-o://ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b" gracePeriod=30 Nov 02 22:29:13.624966 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.624906 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:29:13.741107 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.741065 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd387dc4-be02-428d-90e5-73763b939199-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd387dc4-be02-428d-90e5-73763b939199-openshift-service-ca\") pod \"bd387dc4-be02-428d-90e5-73763b939199\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " Nov 02 22:29:13.741412 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.741388 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls\") pod \"bd387dc4-be02-428d-90e5-73763b939199\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " Nov 02 22:29:13.741601 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.741583 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-metrics-certs\") pod \"bd387dc4-be02-428d-90e5-73763b939199\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " Nov 02 22:29:13.741815 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.741793 4510 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd387dc4-be02-428d-90e5-73763b939199-kube-api-access-2jjpx\" (UniqueName: \"kubernetes.io/projected/bd387dc4-be02-428d-90e5-73763b939199-kube-api-access-2jjpx\") pod \"bd387dc4-be02-428d-90e5-73763b939199\" (UID: \"bd387dc4-be02-428d-90e5-73763b939199\") " Nov 02 22:29:13.743053 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.743025 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd387dc4-be02-428d-90e5-73763b939199-openshift-service-ca" pod "bd387dc4-be02-428d-90e5-73763b939199" (UID: "bd387dc4-be02-428d-90e5-73763b939199"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:13.753616 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.753582 4510 generic.go:356] "Generic (PLEG): container finished" podID="bd387dc4-be02-428d-90e5-73763b939199" containerID="ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b" exitCode=0 Nov 02 22:29:13.753876 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.753629 4510 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" Nov 02 22:29:13.754008 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.753618 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" event={"ID":"bd387dc4-be02-428d-90e5-73763b939199","Type":"ContainerDied","Data":"ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b"} Nov 02 22:29:13.754081 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.754023 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4" event={"ID":"bd387dc4-be02-428d-90e5-73763b939199","Type":"ContainerDied","Data":"2625b02d4f35af6aa71601393a501f9b9ff8525cdcfc70ad22c2873798c533e2"} Nov 02 22:29:13.754142 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.754081 4510 scope.go:122] "RemoveContainer" containerID="ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b" Nov 02 22:29:13.756003 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.755976 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls" pod "bd387dc4-be02-428d-90e5-73763b939199" (UID: "bd387dc4-be02-428d-90e5-73763b939199"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:13.757869 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.757844 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd387dc4-be02-428d-90e5-73763b939199-kube-api-access-2jjpx" pod "bd387dc4-be02-428d-90e5-73763b939199" (UID: "bd387dc4-be02-428d-90e5-73763b939199"). InnerVolumeSpecName "kube-api-access-2jjpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:13.758103 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.758078 4510 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-metrics-certs" pod "bd387dc4-be02-428d-90e5-73763b939199" (UID: "bd387dc4-be02-428d-90e5-73763b939199"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:13.778154 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.778129 4510 scope.go:122] "RemoveContainer" containerID="ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b" Nov 02 22:29:13.778501 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:29:13.778472 4510 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b\": container with ID starting with ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b not found: ID does not exist" containerID="ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b" Nov 02 22:29:13.778576 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.778511 4510 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b"} err="failed to get container status \"ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b\": rpc error: code = NotFound desc = could not find container \"ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b\": container with ID starting with ff2e39afdf10df7616ae9570a86200959aa99ad6165369091bbe79abd6b60c5b not found: ID does not exist" Nov 02 22:29:13.843406 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.843372 4510 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd387dc4-be02-428d-90e5-73763b939199-openshift-service-ca\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:13.843406 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.843404 4510 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-maas-api-tls\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:13.843406 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.843413 4510 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd387dc4-be02-428d-90e5-73763b939199-metrics-certs\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:13.843722 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:13.843422 4510 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjpx\" (UniqueName: \"kubernetes.io/projected/bd387dc4-be02-428d-90e5-73763b939199-kube-api-access-2jjpx\") on node \"ip-10-0-133-33.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:14.166201 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:14.166167 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4"] Nov 02 22:29:14.171827 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:14.171800 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-d9f155-55c4db6f44-zrsp4"] Nov 02 22:29:15.530017 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:29:15.529981 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:29:16.152565 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:16.152508 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd387dc4-be02-428d-90e5-73763b939199" path="/var/lib/kubelet/pods/bd387dc4-be02-428d-90e5-73763b939199/volumes" Nov 02 22:29:18.131331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:18.131295 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:29:25.814694 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:25.814656 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:29:37.509036 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:37.508997 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:29:39.850440 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:39.850392 4510 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-ndztn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:29:39.851093 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:39.851049 4510 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-ndztn" podUID="4696f74b-e8cf-4518-96f7-c6b29e686be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:29:45.535415 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:29:45.535381 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:29:52.042821 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:52.042783 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt"] Nov 02 22:29:52.088507 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:52.088478 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4cxrt"] Nov 02 22:29:52.153968 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:52.153940 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b7be79-9b36-491b-81ca-7093abaf1351" path="/var/lib/kubelet/pods/91b7be79-9b36-491b-81ca-7093abaf1351/volumes" Nov 02 22:29:58.603276 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:29:58.603241 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:30:15.529775 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:30:15.529698 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:30:18.043500 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:18.043468 4510 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t"] Nov 02 22:30:18.089593 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:18.089564 4510 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xhc6t"] Nov 02 22:30:18.152576 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:18.152546 4510 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82165f85-9b80-4522-b1e8-4d13f3b6e6cb" path="/var/lib/kubelet/pods/82165f85-9b80-4522-b1e8-4d13f3b6e6cb/volumes" Nov 02 22:30:31.017551 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:31.017500 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zw95m"] Nov 02 22:30:42.700857 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:42.700829 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-sxccp_e84390d4-96bf-4c9d-90c6-8755afe0bc6b/server/0.log" Nov 02 22:30:42.917713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:42.917686 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5z6g_ad711180-966a-47e7-b085-f0dd00cedbd5/manager/0.log" Nov 02 22:30:43.160624 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:43.160592 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r5kml_f5fbd333-e69a-46ad-9059-0f9adc865b1e/proxy/0.log" Nov 02 22:30:44.208116 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:44.208090 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8455db45c4-n7fsn_a5a190ca-b18a-4c79-a3e9-92dfc23f19e2/maas-api/0.log" Nov 02 22:30:45.099629 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:45.099602 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-49lsd_beb13fd7-9ea5-4d0a-af9b-501b200280a4/postgres/0.log" Nov 02 22:30:45.531877 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:30:45.531840 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:30:47.986175 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:47.986146 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zw95m_79af2810-e0dc-4385-8714-2fd71edead77/limitador/0.log" Nov 02 22:30:49.776023 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:49.775988 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-795b66c7fb-fjd9v_76b51e51-eb64-40a8-beb8-043c9e9d5b51/kube-auth-proxy/0.log" Nov 02 22:30:51.137267 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:51.137234 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-kqfwb_63cbd378-8dec-4cb9-9a52-f145dec09563/istio-proxy/0.log" Nov 02 22:30:51.344937 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:51.344907 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ndztn_4696f74b-e8cf-4518-96f7-c6b29e686be5/istio-proxy/0.log" Nov 02 22:30:51.874974 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:51.874946 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zndhx_4e012812-5cc5-451d-94d6-75adbdc21fff/istio-proxy/0.log" Nov 02 22:30:52.543481 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:52.543454 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lcj5f_9ea1fa27-9f65-44e7-b017-c0837b39a362/payload-pre-processing/0.log" Nov 02 22:30:52.762407 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:52.762381 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jmk5x_eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6/payload-processing/0.log" Nov 02 22:30:54.113217 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:54.113189 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cmhbj_de15bfd8-7fcb-486e-a603-8c4d513dadc9/main/0.log" Nov 02 22:30:54.328713 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:54.328687 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wgtj5_28ac4cf9-d63f-40ee-b6b6-fad22e39c631/main/0.log" Nov 02 22:30:54.553363 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:54.553289 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-w4wp6_ab1fe9ee-732a-4438-8a6d-a6353430059a/main/0.log" Nov 02 22:30:54.772660 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:54.772630 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd_38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0/main/0.log" Nov 02 22:30:54.993373 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:54.993344 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tpgnk_c42cb1c5-171a-4f0b-9607-b16a085ee542/main/0.log" Nov 02 22:30:55.213891 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:55.213859 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-svskm_94522fad-8716-4425-85ed-dba0714e23ad/main/0.log" Nov 02 22:30:57.339826 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:30:57.339798 4510 scope.go:122] "RemoveContainer" containerID="f8e4dcfe423478a846a2ba94301648e21346e169efbbbccb68a05e7a48b2437d" Nov 02 22:31:06.883021 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:06.882993 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-sxccp_e84390d4-96bf-4c9d-90c6-8755afe0bc6b/server/0.log" Nov 02 22:31:07.113851 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:07.113817 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-n5z6g_ad711180-966a-47e7-b085-f0dd00cedbd5/manager/0.log" Nov 02 22:31:07.386106 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:07.386080 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-r5kml_f5fbd333-e69a-46ad-9059-0f9adc865b1e/proxy/0.log" Nov 02 22:31:08.241955 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:08.241921 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8455db45c4-n7fsn_a5a190ca-b18a-4c79-a3e9-92dfc23f19e2/maas-api/0.log" Nov 02 22:31:09.168399 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:09.168368 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-49lsd_beb13fd7-9ea5-4d0a-af9b-501b200280a4/postgres/0.log" Nov 02 22:31:11.654953 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:11.654924 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zw95m_79af2810-e0dc-4385-8714-2fd71edead77/limitador/0.log" Nov 02 22:31:13.209965 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:13.209940 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-795b66c7fb-fjd9v_76b51e51-eb64-40a8-beb8-043c9e9d5b51/kube-auth-proxy/0.log" Nov 02 22:31:14.616946 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:14.616916 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-kqfwb_63cbd378-8dec-4cb9-9a52-f145dec09563/istio-proxy/0.log" Nov 02 22:31:14.842564 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:14.842512 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-ndztn_4696f74b-e8cf-4518-96f7-c6b29e686be5/istio-proxy/0.log" Nov 02 22:31:15.323735 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:15.323706 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-zndhx_4e012812-5cc5-451d-94d6-75adbdc21fff/istio-proxy/0.log" Nov 02 22:31:15.530289 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:31:15.530253 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:31:15.988353 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:15.988323 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-lcj5f_9ea1fa27-9f65-44e7-b017-c0837b39a362/payload-pre-processing/0.log" Nov 02 22:31:16.215067 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:16.215034 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jmk5x_eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6/payload-processing/0.log" Nov 02 22:31:17.328325 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:17.328296 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-cmhbj_de15bfd8-7fcb-486e-a603-8c4d513dadc9/main/0.log" Nov 02 22:31:17.558643 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:17.558611 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-wgtj5_28ac4cf9-d63f-40ee-b6b6-fad22e39c631/main/0.log" Nov 02 22:31:17.778731 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:17.778700 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-w4wp6_ab1fe9ee-732a-4438-8a6d-a6353430059a/main/0.log" Nov 02 22:31:18.001111 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:18.001079 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87qqwd_38cb3dd9-ad4b-47db-a1cb-0dd862d11bf0/main/0.log" Nov 02 22:31:18.222472 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:18.222443 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tpgnk_c42cb1c5-171a-4f0b-9607-b16a085ee542/main/0.log" Nov 02 22:31:18.438877 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:18.438837 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-svskm_94522fad-8716-4425-85ed-dba0714e23ad/main/0.log" Nov 02 22:31:45.531808 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:31:45.531719 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:31:46.639463 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:46.639439 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jmk5x_eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6/payload-processing/0.log" Nov 02 22:31:46.856076 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:46.856042 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jmk5x_eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6/payload-processing/0.log" Nov 02 22:31:57.359431 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:31:57.359401 4510 scope.go:122] "RemoveContainer" containerID="f9bda099bd8605e3bddfb3f56d8f6d1bec3015266c1a6feb99e1ab10e8fad968" Nov 02 22:32:15.530557 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:32:15.530506 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:32:22.692393 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:22.692359 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jmk5x_eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6/payload-processing/0.log" Nov 02 22:32:22.907556 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:22.907513 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jmk5x_eb5bf3d8-3b15-4597-9f54-83bd1faa6ac6/payload-processing/0.log" Nov 02 22:32:31.919089 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:31.919059 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fdpbr_ba030b9f-ec13-45a5-82a9-df4315256051/global-pull-secret-syncer/0.log" Nov 02 22:32:35.766301 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:35.766270 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wcgz4_b08981c9-b9bb-4f2a-aa0e-30a1be35d954/konnectivity-agent/0.log" Nov 02 22:32:36.550123 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:36.550095 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-33.us-west-2.compute.internal_7a52ae9f924fc84a5af3aafcc66e569a/haproxy/0.log" Nov 02 22:32:42.417714 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.417676 4510 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb"] Nov 02 22:32:42.418231 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.418143 4510 cpu_manager.go:414] "containerMap: removing container" podUID="bd387dc4-be02-428d-90e5-73763b939199" containerName="maas-api" Nov 02 22:32:42.418231 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.418155 4510 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd387dc4-be02-428d-90e5-73763b939199" containerName="maas-api" Nov 02 22:32:42.418318 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.418240 4510 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd387dc4-be02-428d-90e5-73763b939199" containerName="maas-api" Nov 02 22:32:42.421477 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.421459 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.457977 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.457947 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qblq\" (UniqueName: \"kubernetes.io/projected/4bd91140-20cd-406d-91e8-de382133f283-kube-api-access-8qblq\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.458099 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.458005 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-sys\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.458165 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.458092 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-lib-modules\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.458219 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.458162 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-proc\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.458273 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.458218 4510 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-podres\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.474788 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.474765 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb"] Nov 02 22:32:42.518579 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.518553 4510 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nvhb8\"/\"default-dockercfg-jq77d\"" Nov 02 22:32:42.518579 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.518571 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nvhb8\"/\"openshift-service-ca.crt\"" Nov 02 22:32:42.518730 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.518576 4510 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nvhb8\"/\"kube-root-ca.crt\"" Nov 02 22:32:42.559307 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559281 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qblq\" (UniqueName: \"kubernetes.io/projected/4bd91140-20cd-406d-91e8-de382133f283-kube-api-access-8qblq\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559413 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559334 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-sys\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559413 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559367 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-lib-modules\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559413 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559409 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-proc\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559545 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559448 4510 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-podres\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559545 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559409 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-sys\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559545 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559486 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-proc\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559545 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559486 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-lib-modules\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.559689 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.559548 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bd91140-20cd-406d-91e8-de382133f283-podres\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.612161 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.612136 4510 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qblq\" (UniqueName: \"kubernetes.io/projected/4bd91140-20cd-406d-91e8-de382133f283-kube-api-access-8qblq\") pod \"perf-node-gather-daemonset-plrjb\" (UID: \"4bd91140-20cd-406d-91e8-de382133f283\") " pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:42.734012 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:42.733947 4510 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:43.103959 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:43.103867 4510 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb"] Nov 02 22:32:43.111699 ip-10-0-133-33 kubenswrapper[4510]: W1102 22:32:43.111665 4510 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bd91140_20cd_406d_91e8_de382133f283.slice/crio-8b0b5ac9a1f9e5dd1751665a82db6d96dc9c4bd03eb5a04ae0bfc960e52c236d WatchSource:0}: Error finding container 8b0b5ac9a1f9e5dd1751665a82db6d96dc9c4bd03eb5a04ae0bfc960e52c236d: Status 404 returned error can't find the container with id 8b0b5ac9a1f9e5dd1751665a82db6d96dc9c4bd03eb5a04ae0bfc960e52c236d Nov 02 22:32:43.941691 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:43.941653 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" event={"ID":"4bd91140-20cd-406d-91e8-de382133f283","Type":"ContainerStarted","Data":"5ea8d50b32b870402039be627ccc2fc2a0ee61a3da920f63ac4ec4f6c565a04a"} Nov 02 22:32:43.942159 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:43.941742 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:43.942159 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:43.941766 4510 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" event={"ID":"4bd91140-20cd-406d-91e8-de382133f283","Type":"ContainerStarted","Data":"8b0b5ac9a1f9e5dd1751665a82db6d96dc9c4bd03eb5a04ae0bfc960e52c236d"} Nov 02 22:32:44.042331 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:44.042303 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zw95m_79af2810-e0dc-4385-8714-2fd71edead77/limitador/0.log" Nov 02 22:32:44.046947 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:44.046905 4510 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" podStartSLOduration=2.046893506 podStartE2EDuration="2.046893506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:32:44.045129297 +0000 UTC m=+4790.021427073" watchObservedRunningTime="2026-11-02 22:32:44.046893506 +0000 UTC m=+4790.023191310" Nov 02 22:32:45.530817 ip-10-0-133-33 kubenswrapper[4510]: E1102 22:32:45.530784 4510 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4c9pl" containerName="extractor" Nov 02 22:32:47.104625 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:47.104591 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ed250005-c680-4d6d-9e28-b99581d03f16/alertmanager/0.log" Nov 02 22:32:47.528312 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:47.528283 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ed250005-c680-4d6d-9e28-b99581d03f16/config-reloader/0.log" Nov 02 22:32:47.950313 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:47.950284 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ed250005-c680-4d6d-9e28-b99581d03f16/kube-rbac-proxy-web/0.log" Nov 02 22:32:48.373501 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:48.373479 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ed250005-c680-4d6d-9e28-b99581d03f16/kube-rbac-proxy/0.log" Nov 02 22:32:48.795247 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:48.795172 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ed250005-c680-4d6d-9e28-b99581d03f16/kube-rbac-proxy-metric/0.log" Nov 02 22:32:49.218454 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:49.218425 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ed250005-c680-4d6d-9e28-b99581d03f16/prom-label-proxy/0.log" Nov 02 22:32:49.640554 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:49.640505 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ed250005-c680-4d6d-9e28-b99581d03f16/init-config-reloader/0.log" Nov 02 22:32:49.957184 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:49.957117 4510 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nvhb8/perf-node-gather-daemonset-plrjb" Nov 02 22:32:50.567106 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:50.567078 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2gspf_018583a9-9602-4e9e-9799-9ae2b89707db/kube-state-metrics/0.log" Nov 02 22:32:50.985421 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:50.985389 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2gspf_018583a9-9602-4e9e-9799-9ae2b89707db/kube-rbac-proxy-main/0.log" Nov 02 22:32:51.405199 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:51.405172 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2gspf_018583a9-9602-4e9e-9799-9ae2b89707db/kube-rbac-proxy-self/0.log" Nov 02 22:32:51.830163 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:51.830089 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b989958c5-kgdg8_8bbdf546-cf6d-4a48-bb1c-ee01e66d2847/metrics-server/0.log" Nov 02 22:32:52.255067 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:52.255035 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-5kmxb_ce4cdffc-6153-440f-99e7-1ba21b4821cc/monitoring-plugin/0.log" Nov 02 22:32:52.683326 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:52.683295 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9p7l_10194992-9c92-47ed-92a6-3075cabbf27d/node-exporter/0.log" Nov 02 22:32:53.103428 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:53.103364 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9p7l_10194992-9c92-47ed-92a6-3075cabbf27d/kube-rbac-proxy/0.log" Nov 02 22:32:53.524091 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:53.524065 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9p7l_10194992-9c92-47ed-92a6-3075cabbf27d/init-textfile/0.log" Nov 02 22:32:56.470915 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:56.470882 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8shvw_2c516d4c-2849-47a2-b48e-83dd60d924d3/kube-rbac-proxy-main/0.log" Nov 02 22:32:56.891804 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:56.891777 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8shvw_2c516d4c-2849-47a2-b48e-83dd60d924d3/kube-rbac-proxy-self/0.log" Nov 02 22:32:57.313215 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:57.313132 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8shvw_2c516d4c-2849-47a2-b48e-83dd60d924d3/openshift-state-metrics/0.log" Nov 02 22:32:57.744972 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:57.744935 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe423ff-907a-403c-b883-5d87efd5750c/prometheus/0.log" Nov 02 22:32:58.165043 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:58.165014 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe423ff-907a-403c-b883-5d87efd5750c/config-reloader/0.log" Nov 02 22:32:58.587915 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:58.587838 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe423ff-907a-403c-b883-5d87efd5750c/thanos-sidecar/0.log" Nov 02 22:32:59.009189 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:59.009158 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe423ff-907a-403c-b883-5d87efd5750c/kube-rbac-proxy-web/0.log" Nov 02 22:32:59.432053 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:59.432026 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe423ff-907a-403c-b883-5d87efd5750c/kube-rbac-proxy/0.log" Nov 02 22:32:59.853264 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:32:59.853190 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe423ff-907a-403c-b883-5d87efd5750c/kube-rbac-proxy-thanos/0.log" Nov 02 22:33:00.275530 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:33:00.275484 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0fe423ff-907a-403c-b883-5d87efd5750c/init-config-reloader/0.log" Nov 02 22:33:03.228568 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:33:03.228537 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bb997b84-t778t_6c270975-023a-4326-9de8-9ad0a11dc490/thanos-query/0.log" Nov 02 22:33:03.648455 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:33:03.648426 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bb997b84-t778t_6c270975-023a-4326-9de8-9ad0a11dc490/kube-rbac-proxy-web/0.log" Nov 02 22:33:04.069596 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:33:04.069504 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bb997b84-t778t_6c270975-023a-4326-9de8-9ad0a11dc490/kube-rbac-proxy/0.log" Nov 02 22:33:04.493878 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:33:04.493853 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bb997b84-t778t_6c270975-023a-4326-9de8-9ad0a11dc490/prom-label-proxy/0.log" Nov 02 22:33:04.914206 ip-10-0-133-33 kubenswrapper[4510]: I1102 22:33:04.914174 4510 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8bb997b84-t778t_6c270975-023a-4326-9de8-9ad0a11dc490/kube-rbac-proxy-rules/0.log"